[2024-09-26 16:00:20.618494] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:20.618495] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-09-26 16:00:20.618497] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-09-26 16:00:20.618498] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-09-26 16:00:20.618500] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-09-26 16:00:20.618501] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:20.618503] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:20.618504] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:20.618505] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:20.618507] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:20.618508] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:20.618510] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:20.618511] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:20.618512] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:20.618514] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:20.618515] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:20.618517] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:20.618518] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:20.618519] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:20.618521] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:20.618522] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:20.618524] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:20.618525] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:20.618526] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:20.618528] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:20.618529] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:20.618531] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:20.618532] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:20.618533] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:20.618535] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:20.618536] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:20.618537] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:20.618539] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:20.618540] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:20.618542] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:20.618543] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:20.618545] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:20.618546] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:20.618547] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:20.618549] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:20.618550] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:20.618551] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:20.618552] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:20.618554] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0) [2024-09-26 16:00:20.618555] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, scheduled_task_count=0) [2024-09-26 16:00:20.618557] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0) [2024-09-26 16:00:20.618558] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, scheduled_task_count=0) [2024-09-26 16:00:20.618559] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0) [2024-09-26 16:00:20.618561] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-09-26 16:00:20.618562] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0) [2024-09-26 16:00:20.618564] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-09-26 16:00:20.618565] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0) [2024-09-26 16:00:20.618566] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, scheduled_task_count=0) [2024-09-26 16:00:20.618568] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_net_count=0) [2024-09-26 16:00:20.618569] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_net_count=0) [2024-09-26 16:00:20.618570] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_net_count=0) [2024-09-26 16:00:20.618572] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_net_count=0) [2024-09-26 16:00:20.618573] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_net_count=0) [2024-09-26 16:00:20.618574] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_net_count=0) [2024-09-26 16:00:20.618575] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_net_count=0) [2024-09-26 16:00:20.618576] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_net_count=0) [2024-09-26 16:00:20.618578] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1991) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=184) [2024-09-26 16:00:20.618585] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620618582, task_id:YB42C0A8002D-0006230056D8C58C-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.618590] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=4] schedule one task(task=0x7f9264d763f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.618596] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337620618595, task_id:YB42C0A8002D-0006230056D8C58D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.618599] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=2] schedule one task(task=0x7f9264d76920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.618602] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337620618601, task_id:YB42C0A8002D-0006230056D8C58E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.618604] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=2] schedule one task(task=0x7f9264d76e50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.618614] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337620618613, task_id:YB42C0A8002D-0006230056D8C58F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.618617] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=3] schedule one task(task=0x7f9264d77380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.618621] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620618620, task_id:YB42C0A8002D-0006230056D8C590-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.618623] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=1] schedule one task(task=0x7f9264d778b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.618627] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337620618627, task_id:YB42C0A8002D-0006230056D8C591-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.618634] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=6] schedule one task(task=0x7f9264d77de0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.618636] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.618651] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=14][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.618654] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726157244792838000) [2024-09-26 16:00:20.618658] INFO [STORAGE] ready_for_flush (ob_tx_data_memtable.cpp:865) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=2] memtable is frozen yet.(this=0x7f92a2294080) [2024-09-26 16:00:20.618659] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.618660] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.618663] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.618665] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.618665] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:20.618667] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:20.618671] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.618672] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.618695] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.618693] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847222}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:20.618698] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.618699] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.618703] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.618705] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=6] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.618707] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861df20, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:502, tablet_addr:[816017-1849-0-2305843642722988032-2](offset=1617920,size=590,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:226}, data_tablet_id:{id:226}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726178585269925002, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726178585269925002, cur_report_version:1726178585269925002, data_checksum:1151028828, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726201924319685000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861e5fc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844296631750656-0](offset=1609728,size=1199,type=2,seq=0), type:"MAJOR", tablet_id:{id:226}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726178585269925002, v:0}}, snapshot_version:1726178585269925002, max_merge_version:1726178585269925002}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337620617712, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcc550, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241777443612000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209404959}, this:0x7f92a3bcc550, timestamp:1727334209404959, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:52, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:27262976}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277811954486000, v:0}, rec_scn:{val:1726241777645865000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843250807218176-2](offset=1613824,size=225,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719032545-0](offset=1614049,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880960381-0](offset=1614205,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447480415-0](offset=1614431,size=193,type=2,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:1726178585269925002}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a73f00, 0x7f92a3bcc550, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.618769] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:00:20.618777] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241777443612000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.618798] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.618803] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.618805] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:20.618808] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.618826] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.618828] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.618829] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:20.618832] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.618834] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:20.618839] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.619372] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:20.619386] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.628154] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726158220122116000) [2024-09-26 16:00:20.628185] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=23] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.629397] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1, storage_schema:{this:0x7f9263a432b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.629417] INFO [STORAGE] prepare_tx_data_list (ob_tx_data_memtable.cpp:266) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=18] construct list more than once. skip this time(this=0x7f92a2294080) [2024-09-26 16:00:20.633732] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.633752] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=19] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337620617654, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:20.633760] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.633857] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=12] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:20.633862] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.633872] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847223}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:20.633896] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:1152921504606847223}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847223}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620618824, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=15.04ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.634158] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=26] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:20.634397] INFO [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:329) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=6] pre-process commit versions row(get_ls_id()={id:1}, do_recycle_=true, current_time=1727337620634395, prev_recycle_time=0) [2024-09-26 16:00:20.634872] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=6] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=7999, total_rows=55993) [2024-09-26 16:00:20.634853] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726163995122514000, tablet={this:0x7f92b862bfe0, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:505, tablet_addr:[816017-1849-0-2305843645944356864-2](offset=1761280,size=593,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:200489}, data_tablet_id:{id:200489}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725991206240736000, v:0}, clog_checkpoint_scn:{val:1726163995200406000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726163995122514000, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1725991205062061000, cur_report_version:1725991205062061000, data_checksum:1210484554, row_count:46137450}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1725120000848336, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726201924319685000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1725120000848336}, table_store_addr:{ptr:{this:0x7f92b862c674, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:4, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305877511761428480-0](offset=1703936,size=32133,type=2,seq=0), type:"MAJOR", tablet_id:{id:200489}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1725991205062061000, v:0}}, snapshot_version:1725991205062061000, max_merge_version:1725991203769806000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305849822107303301-0](offset=1736069,size=6345,type=2,seq=0), type:"MINOR", tablet_id:{id:200489}, scn_range:{start_scn:{val:1725991206240736000, v:0}, end_scn:{val:1726076389100953000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726076389025660000, upper_trans_version:1726076389025660000}, {i:1, addr:[816017-1849-0-2305844368572585550-0](offset=1742414,size=1266,type=2,seq=0), type:"MINI", tablet_id:{id:200489}, scn_range:{start_scn:{val:1726076389100953000, v:0}, end_scn:{val:1726077259218062001, v:0}}, contain_uncommitted_row:"yes", max_merge_version:1726077257041685000, upper_trans_version:1726077259766338000}, {i:2, addr:[816017-1849-0-2305843929412180800-0](offset=1743680,size=857,type=2,seq=0), type:"MINI", tablet_id:{id:200489}, scn_range:{start_scn:{val:1726077259218062001, v:0}, end_scn:{val:1726077600564631000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726077598368094000, upper_trans_version:1726077598368094000}, {i:3, addr:[816017-1849-0-2305849899416723097-0](offset=1744537,size=6417,type=2,seq=0), type:"MINOR", tablet_id:{id:200489}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726163995200406000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726163995200406000, upper_trans_version:1726163995200406000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337620617622, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843710368858112-2](offset=1753088,size=653,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843165981754119-0](offset=1753863,size=146,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255102325657-0](offset=1754009,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1725991205062061000}, medium_info_list:{ptr:(nil), addr:[816017-1849-0-2305843901494906880-0](offset=1757184,size=831,type=2,seq=0)}, auto_inc_seq:{ptr:0x7f92b862cafc, auto_inc_seq:{version:1, intervals_count:1, intervals:{tablet_id:{id:0}, start:55640001, end:9223372036854775807}}, addr:[816017-1849-0-2305843140211950221-0](offset=1753741,size=122,type=2,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1725120001569133000, v:0}, create_commit_version:1725120001569133000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a70080, 0x7f92a1a73730, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.634918] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:20.634930] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.635050] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.635065] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=15] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337620617589, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=18, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:20.635081] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.635290] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:20.635295] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.635306] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200240}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.635312] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C590-0-0, dag_ret:0, dag_status:2, start_time:1727337620618622, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1470, output_block_cnt_arr:0x7f92728f1478, concurrent_cnt:1, estimate_row_cnt:332829, estimate_occupy_size:150994944, latest_update_ts:1727337620618824, estimated_finish_time:1727337637635383}) [2024-09-26 16:00:20.635327] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=14] succeed to init merge ctx(task={this:0x7f9264d778b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C590-0-0, dag_ret:0, dag_status:2, start_time:1727337620618622, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.635334] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=7] task finish process(ret=0, start_time=1727337620618822, end_time=1727337620635334, runtime=16512, *this={this:0x7f9264d778b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C590-0-0, dag_ret:0, dag_status:2, start_time:1727337620618622, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.635329] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:200240}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200240}, table_count:2, [{i:0, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8637b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620618654, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=16.65ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.635385] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.635395] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=10] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337620617639, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:20.635413] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=18] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.635459] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337620617712, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.635472] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752755184, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828752755184, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.635487] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.635491] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.635506] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:20.635509] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.635498] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:226}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752755184, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.635514] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847222}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:20.635519] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:226}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752755184, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:226}, table_count:1, [{i:0, table_key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}, merge_scn:{val:1726241777443612000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620618691, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=16.80ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.635532] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:1152921504606847222}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847222}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620618654, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=16.86ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.635548] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=22] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C591-0-0, dag_ret:0, dag_status:2, start_time:1727337620618633, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}, scanned_row_cnt_arr:0x7f9268515480, output_block_cnt_arr:0x7f9268515488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:56623104, latest_update_ts:1727337620618691, estimated_finish_time:1727337637635575}) [2024-09-26 16:00:20.635556] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d77de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C591-0-0, dag_ret:0, dag_status:2, start_time:1727337620618633, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:20.635563] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=7] task finish process(ret=0, start_time=1727337620618689, end_time=1727337620635563, runtime=16874, *this={this:0x7f9264d77de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C591-0-0, dag_ret:0, dag_status:2, start_time:1727337620618633, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:20.635599] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=35] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:20.635790] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=18] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:20.636102] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58D-0-0, dag_ret:0, dag_status:2, start_time:1727337620618598, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265adb460, output_block_cnt_arr:0x7f9265adb468, concurrent_cnt:1, estimate_row_cnt:85417, estimate_occupy_size:134217728, latest_update_ts:1727337620618654, estimated_finish_time:1727337637636165}) [2024-09-26 16:00:20.636115] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f9264d76920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58D-0-0, dag_ret:0, dag_status:2, start_time:1727337620618598, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.636122] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=7] task finish process(ret=0, start_time=1727337620618651, end_time=1727337620636122, runtime=17471, *this={this:0x7f9264d76920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58D-0-0, dag_ret:0, dag_status:2, start_time:1727337620618598, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.636166] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=2] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.636176] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=5] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.636179] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=2] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.636573] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=3] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=8366, total_rows=58562) [2024-09-26 16:00:20.636656] INFO [STORAGE] get_ls_min_end_scn (ob_ls_tablet_service.cpp:5618) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=8] get ls min end scn finish(ls_checkpoint={val:1726163586617842000, v:0}) [2024-09-26 16:00:20.636666] WDIAG [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:761) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=6][errcode=-4023] failed to get snapshot version(ret=-4023, MTL_ID()=1) [2024-09-26 16:00:20.636671] INFO [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:773) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=4] get tx data recycle scn finish.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, recycle_scn={val:1726076389100953000, v:0}, min_end_scn_from_old_tablets={val:4611686018427387903, v:0}, min_end_scn_from_latest_tablets={val:1726076389100953000, v:0}) [2024-09-26 16:00:20.636677] WDIAG [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:336) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=5][errcode=-4023] get recycle ts failed.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:20.636681] WDIAG [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:236) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=2][errcode=-4023] process commit version row failed.(ret=-4023, ret="OB_EAGAIN", this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337620617526, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:20.636693] INFO [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:254) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=12] [TX DATA MERGE]pre process for merge done(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}, this=0x7f92a2294080, tg=time guard have no click events for optimization, fake_tx_data_guard={tx_data:{tx_id:{txid:9223372036854775807}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337620617526, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:20.636707] WDIAG [STORAGE.COMPACTION] pre_process_tx_data_table_merge_ (ob_tx_table_merge_task.cpp:117) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=13][errcode=-4023] do pre process for tx data table merge failed.(ret=-4023, ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, table={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337620617526, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:20.636715] WDIAG [STORAGE.COMPACTION] inner_init_ctx (ob_tx_table_merge_task.cpp:157) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=7][errcode=-4023] pre process tx data table for merge failed.(ret=-4023, ret="OB_EAGAIN", ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.636718] WDIAG [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1061) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=2][errcode=-4023] fail to inner init ctx(ret=-4023, tablet_id={id:49402}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, create_snapshot_version:0, is_full_merge:false, merge_level:1, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:4, range_array:[], concurrent_cnt:0, is_inited:false}, schema_ctx:{schema_version:1, storage_schema:{this:0x7f9263a432b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:49402}, table_count:1, [{i:0, table_key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref:4}]}, schedule_major:false, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}, merge_scn:{val:4611686018427387903, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:NULL, compaction_filter:NULL, time_guard:total=18.09ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}) [2024-09-26 16:00:20.636736] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:975) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=17][errcode=-4023] failed to build merge ctx(ret=-4023, ctx->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.636739] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:994) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=3][errcode=-4023] sstable merge finish(ret=-4023, ctx=0x7f9263a42060, task={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C58C-0-0, dag_ret:0, dag_status:2, start_time:1727337620618590, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:20.636747] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=7][errcode=-4023] failed to process task(ret=-4023) [2024-09-26 16:00:20.636749] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=1] task finish process(ret=-4023, start_time=1727337620618619, end_time=1727337620636748, runtime=18129, *this={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C58C-0-0, dag_ret:0, dag_status:2, start_time:1727337620618590, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:20.636755] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=5][errcode=-4023] failed to do work(ret=-4023, *task_={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C58C-0-0, dag_ret:0, dag_status:2, start_time:1727337620618590, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:20.636763] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=5] dag finished(dag_ret=-4023, runtime=18173, dag_cnt=45, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264d76030, dag={ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C58C-0-0, dag_ret:-4023, dag_status:5, start_time:1727337620618590, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}) [2024-09-26 16:00:20.636774] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C58C-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337620618582, task_id:YB42C0A8002D-0006230056D8C58C-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.636790] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620636786, task_id:YB42C0A8002D-0006230056D8C592-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.636796] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=5] schedule one task(task=0x7f9264d78310, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.636837] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.636841] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.636844] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.636850] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.636984] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58F-0-0, dag_ret:0, dag_status:2, start_time:1727337620618617, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a45470, output_block_cnt_arr:0x7f9263a45478, concurrent_cnt:1, estimate_row_cnt:329211, estimate_occupy_size:150994944, latest_update_ts:1727337620618654, estimated_finish_time:1727337637637056}) [2024-09-26 16:00:20.636995] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=11] succeed to init merge ctx(task={this:0x7f9264d77380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58F-0-0, dag_ret:0, dag_status:2, start_time:1727337620618617, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.637001] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=6] task finish process(ret=0, start_time=1727337620618650, end_time=1727337620637001, runtime=18351, *this={this:0x7f9264d77380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58F-0-0, dag_ret:0, dag_status:2, start_time:1727337620618617, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.637021] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=2] schedule one task(task=0x7f9287c663c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.637087] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c74060, count:4, allocator:0x7f928643e7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9262c74350, local_datums:0x7f9262c74350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:20.637088] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92a9a56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665ae060, count:4, allocator:0x7f92636627d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665ae350, local_datums:0x7f92665ae350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:20.637111] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=21] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684e6060, count:4, allocator:0x7f928643f558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684e6350, local_datums:0x7f92684e6350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:20.637118] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=25] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92a9a56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268474060, count:4, allocator:0x7f9263663558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9268474350, local_datums:0x7f9268474350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:20.637365] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:318) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=8] successfully alloc block(macro_id=[1091](ver=0,mode=0,seq=824626)) [2024-09-26 16:00:20.637372] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:318) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=8] successfully alloc block(macro_id=[1812](ver=0,mode=0,seq=824627)) [2024-09-26 16:00:20.637659] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a68060, count:3, allocator:0x7f92691367d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a68320, local_datums:0x7f9264a68320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:20.637676] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=15] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d38060, count:3, allocator:0x7f9269137558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287d38320, local_datums:0x7f9287d38320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:20.637884] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=9][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:20.637962] ERROR issue_dba_error (ob_log.cpp:1875) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.637967] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf2041c8 0xf217c49 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.637977] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=9][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:20.637980] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:20.637983] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9287dd8108, hex: 313030303030333830, cstr: 100000380},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd80a8, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd80e0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-29413, column_count:20, rowkey_column_count:3, has_column_checksum:0, row_count:100, row_store_type:0, opt:0, var_column_count:0, row_offset:16082, original_length:16422, max_merged_trans_version:1726189672100745000, data_length:16454, data_zlength:5272, data_checksum:4171874375, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9265b00030, buf_size:5272, data_size:16454, row_count:100, column_count:20, max_merged_trans_version:1726189672100745000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:371002229, row_count_delta:22, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16422}) [2024-09-26 16:00:20.638000] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=17][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:20.638002] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:20.638004] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:20.638006] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:20.638008] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1526) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, partition_fuser_->get_result_row()={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a66148, local_datums:0x7f9264a66148}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66190, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a661c8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a663c0, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a663f8, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, fuse_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a68320, local_datums:0x7f9264a68320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a683a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a685d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9269136118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269136118, limit_param:null, stmt_allocator:0x7f92691367d8, allocator:0x7f92691367d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a68320, local_datums:0x7f9264a68320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a683a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a685d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287d38320, local_datums:0x7f9287d38320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9269136e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269136e98, limit_param:null, stmt_allocator:0x7f9269137558, allocator:0x7f9269137558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287d38320, local_datums:0x7f9287d38320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337620617589, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:20.638137] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=129][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a68320, local_datums:0x7f9264a68320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a683a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a685d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9269136118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269136118, limit_param:null, stmt_allocator:0x7f92691367d8, allocator:0x7f92691367d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a68320, local_datums:0x7f9264a68320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a683a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a685d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287d38320, local_datums:0x7f9287d38320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9269136e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269136e98, limit_param:null, stmt_allocator:0x7f9269137558, allocator:0x7f9269137558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287d38320, local_datums:0x7f9287d38320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337620617589, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:20.638212] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=75][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:20.638215] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=2] partition merge iter row count(i=0, row_count=23, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.638228] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=9] partition merge iter row count(i=1, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337620617589, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.638237] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:20.638250] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:20.638253] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:20.638255] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337620637604, end_time=1727337620638255, runtime=651, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58D-0-0, dag_ret:0, dag_status:2, start_time:1727337620618598, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.638262] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58D-0-0, dag_ret:0, dag_status:2, start_time:1727337620618598, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:20.638270] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=19671, dag_cnt=44, dag_cnts_[dag.get_type()]=44, &dag=0x7f9264d76560, dag={ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620618598, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.638279] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58D-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620618595, task_id:YB42C0A8002D-0006230056D8C58D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.638359] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=226, range=0x7f92aae56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:5, allocator:0x7f92636147d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:20.644706] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=13][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:20.644763] ERROR issue_dba_error (ob_log.cpp:1875) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.644774] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.644800] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=26][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:20.644803] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:20.644805] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d620a8, hex: 0000000000000000, int: 0},idx=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9264d620e0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62118, hex: 58A3C56A4A280BE8, int: -1726241732082425000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-19590, column_count:10, rowkey_column_count:5, has_column_checksum:0, row_count:224, row_store_type:0, opt:0, var_column_count:0, row_offset:15527, original_length:16363, max_merged_trans_version:1726241777443612000, data_length:16363, data_zlength:16363, data_checksum:2057046089, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926fa04090, buf_size:16363, data_size:16363, row_count:224, column_count:10, max_merged_trans_version:1726241777443612000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1006468742, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16363}) [2024-09-26 16:00:20.644820] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=14][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:20.644822] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:20.644824] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:20.644825] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:20.644827] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:20.644829] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:20.644831] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263b5e3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263b5e3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f9263614118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263614118, limit_param:null, stmt_allocator:0x7f92636147d8, allocator:0x7f92636147d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263b5e3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337620617712, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}) [2024-09-26 16:00:20.644894] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=63][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:20.644882] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b863bfe0, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:506, tablet_addr:[816017-1849-0-2305843647017062400-1](offset=724992,size=594,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, data_tablet_id:{id:200489}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725991206240736000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1725991205062061000, cur_report_version:1725991205062061000, data_checksum:2143506547, row_count:46137450}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1725120000848352, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1725120000848352}, table_store_addr:{ptr:null, addr:[816017-1849-0-2305845196426506240-1](offset=716800,size=2037,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718137333-0](offset=718837,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255101290641-0](offset=718993,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1725991205062061000}, medium_info_list:{ptr:(nil), addr:[816017-1849-0-2305843520315523072-0](offset=720896,size=476,type=2,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1725120001569133000, v:0}, create_commit_version:1725120001569133000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a746d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:20.644895] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263b5e3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f9263614118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263614118, limit_param:null, stmt_allocator:0x7f92636147d8, allocator:0x7f92636147d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263b5e3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337620617712, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}]) [2024-09-26 16:00:20.644929] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=33][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:20.644929] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=46] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:20.644931] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=1] partition merge iter row count(i=0, row_count=224, ghost_row_count=0, pkey={tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, table={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337620617712, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.644943] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:20.644947] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.644955] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:20.644958] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:20.644960] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337620638310, end_time=1727337620644960, runtime=6650, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C591-0-0, dag_ret:0, dag_status:2, start_time:1727337620618633, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:20.644976] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C591-0-0, dag_ret:0, dag_status:2, start_time:1727337620618633, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:20.644986] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=7] dag finished(dag_ret=-4184, runtime=26353, dag_cnt=43, dag_cnts_[dag.get_type()]=43, &dag=0x7f9264d77a20, dag={ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C591-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620618633, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}) [2024-09-26 16:00:20.644995] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C591-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620618627, task_id:YB42C0A8002D-0006230056D8C591-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.645134] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337620645129, task_id:YB42C0A8002D-0006230056D8C593-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.645143] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=7] schedule one task(task=0x7f9264d78840, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.645155] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337620645154, task_id:YB42C0A8002D-0006230056D8C594-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.645159] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=4] schedule one task(task=0x7f9264d78d70, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.645807] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.645816] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=8][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.645819] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.645827] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.645852] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=19][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.645857] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.645859] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.645862] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.645865] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.645876] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.648943] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.648960] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=17] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337620617622, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=17, max_schema_version_on_memtable=1725120002010696) [2024-09-26 16:00:20.648968] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=17, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.649071] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1887, total_rows=92463) [2024-09-26 16:00:20.649075] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.649084] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200489}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:19, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.649106] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:200489}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200489}, table_count:2, [{i:0, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620618796, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=30.28ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.649472] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:20.649479] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.649497] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=24] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58E-0-0, dag_ret:0, dag_status:2, start_time:1727337620618604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab460, output_block_cnt_arr:0x7f92684ab468, concurrent_cnt:1, estimate_row_cnt:254187, estimate_occupy_size:138412032, latest_update_ts:1727337620618796, estimated_finish_time:1727337637649562}) [2024-09-26 16:00:20.649509] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=12] succeed to init merge ctx(task={this:0x7f9264d76e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58E-0-0, dag_ret:0, dag_status:2, start_time:1727337620618604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.649516] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=6] task finish process(ret=0, start_time=1727337620618794, end_time=1727337620649515, runtime=30721, *this={this:0x7f9264d76e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58E-0-0, dag_ret:0, dag_status:2, start_time:1727337620618604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.649528] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:20.649538] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=2] schedule one task(task=0x7f9287c66750, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.649547] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=14] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.649804] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d32060, count:3, allocator:0x7f92636147d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9287d32320, local_datums:0x7f9287d32320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:20.649825] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=18] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:3, allocator:0x7f9263615558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:20.649982] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=7][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:20.650057] ERROR issue_dba_error (ob_log.cpp:1875) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.650061] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.650067] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=5][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:20.650070] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:20.650072] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd2070, hex: 9F094D0300000000, int: 55380383},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd20a8, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd20e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3044, column_count:19, rowkey_column_count:3, has_column_checksum:0, row_count:83, row_store_type:0, opt:5, var_column_count:0, row_offset:16109, original_length:16381, max_merged_trans_version:1726163995623997000, data_length:16381, data_zlength:7759, data_checksum:3416849541, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265a74030, buf_size:7759, data_size:16381, row_count:83, column_count:19, max_merged_trans_version:1726163995623997000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3537734793, row_count_delta:83, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16381}) [2024-09-26 16:00:20.650085] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=12][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:20.650087] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:20.650088] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:20.650090] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:20.650092] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a624f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a625d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a626e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a624f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a625d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a626e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f9263614e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263614e98, limit_param:null, stmt_allocator:0x7f9263615558, allocator:0x7f9263615558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a624f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a625d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a626e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337620617622, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}) [2024-09-26 16:00:20.650160] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=68][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:20.650162] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a624f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a625d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a626e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f9263614e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263614e98, limit_param:null, stmt_allocator:0x7f9263615558, allocator:0x7f9263615558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a624f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a625d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a626e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337620617622, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}]) [2024-09-26 16:00:20.650200] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=38][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:20.650203] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=2] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.650216] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=9] partition merge iter row count(i=1, row_count=83, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, table={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337620617622, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.650225] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:20.650236] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:20.650240] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:20.650242] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337620649737, end_time=1727337620650242, runtime=505, *this={this:0x7f9287c66750, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58E-0-0, dag_ret:0, dag_status:2, start_time:1727337620618604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.650249] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287c66750, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58E-0-0, dag_ret:0, dag_status:2, start_time:1727337620618604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:20.650258] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=31653, dag_cnt=42, dag_cnts_[dag.get_type()]=42, &dag=0x7f9264d76a90, dag={ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620618604, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.650267] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58E-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620618601, task_id:YB42C0A8002D-0006230056D8C58E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.650285] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337620650282, task_id:YB42C0A8002D-0006230056D8C595-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.650291] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=6] schedule one task(task=0x7f9264d792a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.650322] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337620617728, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.650334] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337620617747, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.650344] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=9] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:20.650343] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=20] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:20.650347] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=4] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9287db9480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.650347] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9264ddb480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.650464] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3214, total_rows=22498) [2024-09-26 16:00:20.650467] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.650467] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3223, total_rows=22561) [2024-09-26 16:00:20.650471] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.650473] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847408}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.650477] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264ddb480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847409}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.650492] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:1152921504606847408}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9287db9480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847408}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620636826, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=13.64ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.650497] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:1152921504606847409}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9264ddb480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847409}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8640e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620645796, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.68ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.650608] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C592-0-0, dag_ret:0, dag_status:2, start_time:1727337620636795, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db95d0, output_block_cnt_arr:0x7f9287db95d8, concurrent_cnt:1, estimate_row_cnt:35098, estimate_occupy_size:18874368, latest_update_ts:1727337620636826, estimated_finish_time:1727337637650617}) [2024-09-26 16:00:20.650614] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C593-0-0, dag_ret:0, dag_status:2, start_time:1727337620645142, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264ddb5d0, output_block_cnt_arr:0x7f9264ddb5d8, concurrent_cnt:1, estimate_row_cnt:32439, estimate_occupy_size:18874368, latest_update_ts:1727337620645796, estimated_finish_time:1727337637650623}) [2024-09-26 16:00:20.650617] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d78310, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C592-0-0, dag_ret:0, dag_status:2, start_time:1727337620636795, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.650621] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d78840, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C593-0-0, dag_ret:0, dag_status:2, start_time:1727337620645142, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.650622] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=5] task finish process(ret=0, start_time=1727337620636823, end_time=1727337620650622, runtime=13799, *this={this:0x7f9264d78310, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C592-0-0, dag_ret:0, dag_status:2, start_time:1727337620636795, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.650627] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=5] task finish process(ret=0, start_time=1727337620645792, end_time=1727337620650626, runtime=4834, *this={this:0x7f9264d78840, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C593-0-0, dag_ret:0, dag_status:2, start_time:1727337620645142, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.650666] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=17] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.650673] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=4] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.650972] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.650979] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.650982] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.650988] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.650994] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.651003] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241621541056000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.651034] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:20.651051] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337620617762, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.651062] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828761605424, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828761605424, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.651073] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.651077] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.651082] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200004}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1710828761605424, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.651099] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:200004}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828761605424, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200004}, table_count:1, [{i:0, table_key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8621da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620645848, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.23ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.651120] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C594-0-0, dag_ret:0, dag_status:2, start_time:1727337620645159, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265adb460, output_block_cnt_arr:0x7f9265adb468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337620645848, estimated_finish_time:1727337637651123}) [2024-09-26 16:00:20.651127] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d78d70, type:15, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C594-0-0, dag_ret:0, dag_status:2, start_time:1727337620645159, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.651132] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=5] task finish process(ret=0, start_time=1727337620645847, end_time=1727337620651132, runtime=5285, *this={this:0x7f9264d78d70, type:15, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C594-0-0, dag_ret:0, dag_status:2, start_time:1727337620645159, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.651149] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=3] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.651384] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337620617777, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.651396] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:0, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.651409] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.651412] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.651418] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:0, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.651435] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:1}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:0, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1}, table_count:1, [{i:0, table_key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}, merge_scn:{val:1726241621541056000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620650964, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=451us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.651460] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C595-0-0, dag_ret:0, dag_status:2, start_time:1727337620650291, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d480, output_block_cnt_arr:0x7f9264b1d488, concurrent_cnt:1, estimate_row_cnt:3, estimate_occupy_size:6291456, latest_update_ts:1727337620650964, estimated_finish_time:1727337637651462}) [2024-09-26 16:00:20.651483] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=22] succeed to init merge ctx(task={this:0x7f9264d792a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C595-0-0, dag_ret:0, dag_status:2, start_time:1727337620650291, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:20.651489] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=5] task finish process(ret=0, start_time=1727337620650961, end_time=1727337620651488, runtime=527, *this={this:0x7f9264d792a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C595-0-0, dag_ret:0, dag_status:2, start_time:1727337620650291, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:20.651506] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=3] schedule one task(task=0x7f9287c66750, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.651661] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1, range=0x7f92a98d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:8}, cols_desc:{ObIArray:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d34060, count:5, allocator:0x7f92c1d9e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:20.652026] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:20.652072] ERROR issue_dba_error (ob_log.cpp:1875) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.652076] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.652084] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:20.652087] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:20.652089] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 17, flag: 0, null: 0, ptr: 0x7f9287d38178, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d380a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 15, flag: 0, null: 0, ptr: 0x7f9287d38189, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38118, hex: 60D2640CCC290BE8, int: -1726240075808452000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-20032, column_count:8, rowkey_column_count:5, has_column_checksum:0, row_count:158, row_store_type:0, opt:0, var_column_count:0, row_offset:15849, original_length:16421, max_merged_trans_version:1726241621541056000, data_length:16421, data_zlength:16421, data_checksum:3599888316, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926fc04090, buf_size:16421, data_size:16421, row_count:158, column_count:8, max_merged_trans_version:1726241621541056000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1714289658, row_count_delta:2, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16421}) [2024-09-26 16:00:20.652101] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:20.652103] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=1][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:20.652105] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:20.652106] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:20.652108] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:20.652109] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:20.652111] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d343a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d343a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f92c1d9e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c1d9e118, limit_param:null, stmt_allocator:0x7f92c1d9e7d8, allocator:0x7f92c1d9e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d343a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337620617777, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:20.652186] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=75][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:20.652189] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d343a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f92c1d9e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c1d9e118, limit_param:null, stmt_allocator:0x7f92c1d9e7d8, allocator:0x7f92c1d9e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d343a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337620617777, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:20.652227] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=37][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:20.652229] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=1] partition merge iter row count(i=0, row_count=158, ghost_row_count=0, pkey={tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, table={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337620617777, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.652242] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:20.652251] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:20.652255] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:20.652257] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337620651610, end_time=1727337620652257, runtime=647, *this={this:0x7f9287c66750, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C595-0-0, dag_ret:0, dag_status:2, start_time:1727337620650291, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:20.652263] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287c66750, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C595-0-0, dag_ret:0, dag_status:2, start_time:1727337620650291, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:20.652271] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=1980, dag_cnt=41, dag_cnts_[dag.get_type()]=41, &dag=0x7f9264d78ee0, dag={ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C595-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620650291, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}) [2024-09-26 16:00:20.652279] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C595-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620650282, task_id:YB42C0A8002D-0006230056D8C595-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.652296] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620652293, task_id:YB42C0A8002D-0006230056D8C596-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.652301] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=4] schedule one task(task=0x7f9264d797d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.652340] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=200004, range=0x7f92a9ed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d2e060, count:3, allocator:0x7f92a31fe7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:20.652503] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=11][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:20.652541] ERROR issue_dba_error (ob_log.cpp:1875) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.652544] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.652550] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:20.652552] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:20.652557] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=5][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dce070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dce0a8, hex: C852A9F488570BE8, int: -1726189786434547000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dce0e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19071, column_count:8, rowkey_column_count:3, has_column_checksum:0, row_count:218, row_store_type:0, opt:0, var_column_count:0, row_offset:15549, original_length:16361, max_merged_trans_version:1726190661653507000, data_length:16361, data_zlength:5996, data_checksum:2604580500, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9287dd0030, buf_size:5996, data_size:16361, row_count:218, column_count:8, max_merged_trans_version:1726190661653507000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:83488362, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16361}) [2024-09-26 16:00:20.652589] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=32][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:20.652592] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=3][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:20.652594] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:20.652597] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:20.652598] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f92a31fe118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31fe118, limit_param:null, stmt_allocator:0x7f92a31fe7d8, allocator:0x7f92a31fe7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337620617762, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:20.652653] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=55][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:20.652655] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f92a31fe118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31fe118, limit_param:null, stmt_allocator:0x7f92a31fe7d8, allocator:0x7f92a31fe7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337620617762, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:20.652689] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=33][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:20.652690] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=1] partition merge iter row count(i=0, row_count=218, ghost_row_count=0, pkey={tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337620617762, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.652702] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:20.652710] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:20.652714] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:20.652715] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337620652305, end_time=1727337620652715, runtime=410, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C594-0-0, dag_ret:0, dag_status:2, start_time:1727337620645159, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.652721] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C594-0-0, dag_ret:0, dag_status:2, start_time:1727337620645159, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:20.652728] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=7569, dag_cnt=40, dag_cnts_[dag.get_type()]=40, &dag=0x7f9264d789b0, dag={ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C594-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620645159, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.652736] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C594-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620645154, task_id:YB42C0A8002D-0006230056D8C594-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.652750] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620652748, task_id:YB42C0A8002D-0006230056D8C597-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.652753] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=3] schedule one task(task=0x7f9264d79d00, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.652788] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1152921504606847408, range=0x7f92aa352200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d34060, count:5, allocator:0x7f92c19447d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:20.653006] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=11][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:20.653040] ERROR issue_dba_error (ob_log.cpp:1875) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.653043] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.653049] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:20.653051] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:20.653056] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=5][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9287d38178, hex: 30323232362D32312D373825, cstr: 02226-21-78%},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9287d38184, hex: 3230323430393133303435343039, cstr: 20240913045409},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d380e0, hex: CD7F500300000000, int: 55607245},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38118, hex: E8EC05CCA0600BE8, int: -1726179788432479000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-11869, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:238, row_store_type:0, opt:5, var_column_count:0, row_offset:15429, original_length:16321, max_merged_trans_version:1726190643638412000, data_length:16321, data_zlength:5388, data_checksum:152456761, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265b00030, buf_size:5388, data_size:16321, row_count:238, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2114797869, row_count_delta:238, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16321}) [2024-09-26 16:00:20.653067] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:20.653068] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:20.653070] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:20.653072] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:20.653073] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d343e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d343e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92c1944118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c1944118, limit_param:null, stmt_allocator:0x7f92c19447d8, allocator:0x7f92c19447d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d343e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337620617728, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:20.653118] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=45][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:20.653119] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d343e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92c1944118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c1944118, limit_param:null, stmt_allocator:0x7f92c19447d8, allocator:0x7f92c19447d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d343e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337620617728, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:20.653150] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=30][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:20.653151] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=1] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337620617728, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.653161] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:20.653169] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:20.653172] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=2][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:20.653173] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337620652760, end_time=1727337620653173, runtime=413, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C592-0-0, dag_ret:0, dag_status:2, start_time:1727337620636795, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.653179] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C592-0-0, dag_ret:0, dag_status:2, start_time:1727337620636795, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:20.653186] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=16391, dag_cnt=39, dag_cnts_[dag.get_type()]=39, &dag=0x7f9264d77f50, dag={ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C592-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620636795, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.653193] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C592-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620636786, task_id:YB42C0A8002D-0006230056D8C592-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.653206] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620653204, task_id:YB42C0A8002D-0006230056D8C598-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.653209] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=2] schedule one task(task=0x7f9264d7a230, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.653254] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=1152921504606847409, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5a060, count:5, allocator:0x7f926912c7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9264d5a360, local_datums:0x7f9264d5a360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:20.653460] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=11][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:20.653493] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.653496] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.653503] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:20.653505] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:20.653507] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 7, flag: 0, null: 0, ptr: 0x7f9264d62178, hex: 31303033343330, cstr: 1003430},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9264d6217f, hex: 3230323430393133303435303537, cstr: 20240913045057},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d620e0, hex: 6E48500300000000, int: 55593070},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62118, hex: 58B03BC5A1610BE8, int: -1726178684739801000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-26357, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:250, row_store_type:0, opt:5, var_column_count:0, row_offset:15394, original_length:16334, max_merged_trans_version:1726190643638412000, data_length:16334, data_zlength:5786, data_checksum:2846467531, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9263bde030, buf_size:5786, data_size:16334, row_count:250, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:796410710, row_count_delta:250, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16334}) [2024-09-26 16:00:20.653519] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=12][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:20.653521] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:20.653525] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=4][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:20.653527] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:20.653529] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9264d5a360, local_datums:0x7f9264d5a360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9264d5a360, local_datums:0x7f9264d5a360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f926912c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926912c118, limit_param:null, stmt_allocator:0x7f926912c7d8, allocator:0x7f926912c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320133, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9264d5a360, local_datums:0x7f9264d5a360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337620617747, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:20.653597] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=66][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:20.653599] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9264d5a360, local_datums:0x7f9264d5a360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f926912c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926912c118, limit_param:null, stmt_allocator:0x7f926912c7d8, allocator:0x7f926912c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320133, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9264d5a360, local_datums:0x7f9264d5a360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337620617747, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:20.653631] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=32][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:20.653633] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=2] partition merge iter row count(i=0, row_count=250, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337620617747, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.653644] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:20.653653] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:20.653657] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:20.653658] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337620653215, end_time=1727337620653658, runtime=443, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C593-0-0, dag_ret:0, dag_status:2, start_time:1727337620645142, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.653664] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C593-0-0, dag_ret:0, dag_status:2, start_time:1727337620645142, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:20.653671] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=8529, dag_cnt=38, dag_cnts_[dag.get_type()]=38, &dag=0x7f9264d78480, dag={ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C593-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620645142, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.653678] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C593-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620645129, task_id:YB42C0A8002D-0006230056D8C593-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.653692] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620653691, task_id:YB42C0A8002D-0006230056D8C599-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.653696] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=3] schedule one task(task=0x7f9264d7a760, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.654256] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.654265] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=8][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.654268] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.654273] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.654277] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.654286] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241465393532000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.654524] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337620617834, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.654547] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=22] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.654562] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.654566] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.654573] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:331}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.654592] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:331}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:331}, table_count:1, [{i:0, table_key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}, merge_scn:{val:1726241465393532000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8633ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620654249, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=322us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.654622] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C599-0-0, dag_ret:0, dag_status:2, start_time:1727337620653696, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}, scanned_row_cnt_arr:0x7f9263a39480, output_block_cnt_arr:0x7f9263a39488, concurrent_cnt:1, estimate_row_cnt:45, estimate_occupy_size:0, latest_update_ts:1727337620654249, estimated_finish_time:1727337637654621}) [2024-09-26 16:00:20.654629] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7a760, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C599-0-0, dag_ret:0, dag_status:2, start_time:1727337620653696, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:20.654635] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=6] task finish process(ret=0, start_time=1727337620654233, end_time=1727337620654635, runtime=402, *this={this:0x7f9264d7a760, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C599-0-0, dag_ret:0, dag_status:2, start_time:1727337620653696, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:20.654654] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=2] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.654792] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=331, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268434060, count:5, allocator:0x7f926912c7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9268434360, local_datums:0x7f9268434360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:20.654997] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=16][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:20.655035] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.655038] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.655046] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:20.655049] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:20.655052] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9268592070, hex: 0000000000000000, int: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685920a8, hex: 3501000000000000, int: 309},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685920e0, hex: 3501000000000000, int: 309},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9268592118, hex: 7032CD318A310BE8, int: -1726231562555674000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9268592150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-24587, column_count:14, rowkey_column_count:5, has_column_checksum:0, row_count:196, row_store_type:0, opt:0, var_column_count:0, row_offset:15642, original_length:16366, max_merged_trans_version:1726241465393532000, data_length:16366, data_zlength:16366, data_checksum:3184742917, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b7004090, buf_size:16366, data_size:16366, row_count:196, column_count:14, max_merged_trans_version:1726241465393532000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2306482841, row_count_delta:7, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16366}) [2024-09-26 16:00:20.655063] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=11][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:20.655065] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:20.655067] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:20.655068] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:20.655070] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:20.655072] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:20.655074] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9268434360, local_datums:0x7f9268434360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684343a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684343e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f92684344f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92684345a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684345d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:331}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752994696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9268434360, local_datums:0x7f9268434360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684343a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684343e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f92684344f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92684345a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684345d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, store_ctx:{this:0x7f926912c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926912c118, limit_param:null, stmt_allocator:0x7f926912c7d8, allocator:0x7f926912c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241465393532000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3906362710315644976, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9268434360, local_datums:0x7f9268434360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684343a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684343e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f92684344f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92684345a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684345d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, key:{"BIGINT":0},{"BIGINT":309},{"BIGINT":309}, value_iter:{value:{this=0x7f92830f5dd8 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=15 list_head=0x7f928848eeb8 latest_compact_node=0x7f9283113eb0 max_trans_version={val:1726241465393532000, v:0} max_trans_id=125582299 max_elr_trans_version={val:1726241465393532000, v:0} max_elr_trans_id=125582299 latest_compact_ts=1727334207881700 last_compact_cnt=49 total_trans_node_cnt=141 max_modify_scn={val:1726241465393532000, v:0} min_modify_scn={val:1726179340987317000, v:0}}, version_iter:this=0x7f928848eb78 trans_version={val:1726241465393532000, v:0} scn={val:1726241465393532000, v:0} tx_id={txid:125582299} prev=0x7f928848e850 next=0x7f928848eeb8 modify_count=65 acc_checksum=2418483105 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:42} seq_no=1726241465159151, multi_version_iter:this=0x7f9281a21200 trans_version={val:1726230662165479000, v:0} scn={val:1726230662165479000, v:0} tx_id={txid:125497658} prev=0x7f9281a20ec0 next=0x7f9283111860 modify_count=41 acc_checksum=883883390 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:76} seq_no=1726230662033825, max_committed_trans_version:1726241465393532000, cur_trans_version:{val:1726231562555674000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241465393532000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:196, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337620617834, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:14, cur_pos:0, count:0}}) [2024-09-26 16:00:20.655157] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=83][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:20.655159] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:331}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752994696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9268434360, local_datums:0x7f9268434360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684343a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684343e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f92684344f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92684345a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684345d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, store_ctx:{this:0x7f926912c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926912c118, limit_param:null, stmt_allocator:0x7f926912c7d8, allocator:0x7f926912c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241465393532000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3906362710315644976, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9268434360, local_datums:0x7f9268434360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684343a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684343e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f92684344f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92684345a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684345d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9268434610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, key:{"BIGINT":0},{"BIGINT":309},{"BIGINT":309}, value_iter:{value:{this=0x7f92830f5dd8 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=15 list_head=0x7f928848eeb8 latest_compact_node=0x7f9283113eb0 max_trans_version={val:1726241465393532000, v:0} max_trans_id=125582299 max_elr_trans_version={val:1726241465393532000, v:0} max_elr_trans_id=125582299 latest_compact_ts=1727334207881700 last_compact_cnt=49 total_trans_node_cnt=141 max_modify_scn={val:1726241465393532000, v:0} min_modify_scn={val:1726179340987317000, v:0}}, version_iter:this=0x7f928848eb78 trans_version={val:1726241465393532000, v:0} scn={val:1726241465393532000, v:0} tx_id={txid:125582299} prev=0x7f928848e850 next=0x7f928848eeb8 modify_count=65 acc_checksum=2418483105 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:42} seq_no=1726241465159151, multi_version_iter:this=0x7f9281a21200 trans_version={val:1726230662165479000, v:0} scn={val:1726230662165479000, v:0} tx_id={txid:125497658} prev=0x7f9281a20ec0 next=0x7f9283111860 modify_count=41 acc_checksum=883883390 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:76} seq_no=1726230662033825, max_committed_trans_version:1726241465393532000, cur_trans_version:{val:1726231562555674000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241465393532000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:196, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337620617834, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:14, cur_pos:0, count:0}}]) [2024-09-26 16:00:20.655200] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=41][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:20.655202] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=1] partition merge iter row count(i=0, row_count=196, ghost_row_count=0, pkey={tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, table={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337620617834, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.655213] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:20.655223] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:20.655227] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:20.655229] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337620654748, end_time=1727337620655228, runtime=480, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C599-0-0, dag_ret:0, dag_status:2, start_time:1727337620653696, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:20.655235] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C599-0-0, dag_ret:0, dag_status:2, start_time:1727337620653696, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:20.655243] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=1546, dag_cnt=37, dag_cnts_[dag.get_type()]=37, &dag=0x7f9264d7a3a0, dag={ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C599-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620653696, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}) [2024-09-26 16:00:20.655251] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C599-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620653691, task_id:YB42C0A8002D-0006230056D8C599-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.655268] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337620655265, task_id:YB42C0A8002D-0006230056D8C59A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.655273] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=3] schedule one task(task=0x7f9264d7ac90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.655284] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.655287] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.655309] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=21][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.655316] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.655321] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8631da0, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:515, tablet_addr:[816017-1849-0-2305843656681984000-2](offset=1970176,size=603,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:309}, data_tablet_id:{id:309}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:810117565, row_count:8}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86324c4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844526412836864-0](offset=1945600,size=1413,type=2,seq=0), type:"MAJOR", tablet_id:{id:309}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077442448649001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843990615668101-0](offset=1947013,size=914,type=2,seq=0), type:"MINOR", tablet_id:{id:309}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178572306341000, upper_trans_version:1726178572306341000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337620617820, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcdcc0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241316899724000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405581}, this:0x7f92a3bcdcc0, timestamp:1727334209405581, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:51, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277651138194000, v:0}, rec_scn:{val:1726241616921722000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697937408-2](offset=1957888,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843189604278604-0](offset=1958220,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881304564-0](offset=1958388,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447824598-0](offset=1958614,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a76de0, 0x7f92a3bcdcc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.655366] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=45] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.655373] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241316899724000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.655395] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.655408] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=12][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.655410] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.655414] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.655416] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8632d40, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[816017-1849-0-2305843650239590400-2](offset=2027520,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:444}, data_tablet_id:{id:444}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1509715702, row_count:4}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8633440, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844406153818112-0](offset=2011136,size=1301,type=2,seq=0), type:"MAJOR", tablet_id:{id:444}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077198228842001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843949813544213-0](offset=2012437,size=876,type=2,seq=0), type:"MINOR", tablet_id:{id:444}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726177998310996000, upper_trans_version:1726177998310996000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337620617805, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bce490, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209461185}, this:0x7f92a3bce490, timestamp:1727334209461185, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:54, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277298911712000, v:0}, rec_scn:{val:1726241718316746000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697998848-2](offset=2019328,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161889100-0](offset=2019660,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881365998-0](offset=2019822,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447886032-0](offset=2020048,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a775b0, 0x7f92a3bce490, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.655448] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=31] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.655452] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.655487] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.655492] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.655494] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.655496] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.655499] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726176799210367000, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b862fe60, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:497, tablet_addr:[816017-1849-0-2305843637353291776-2](offset=630784,size=585,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:412}, data_tablet_id:{id:412}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726176799210367000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3559841642, row_count:4}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8630518, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844255828574208-0](offset=622592,size=1161,type=2,seq=0), type:"MAJOR", tablet_id:{id:412}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725843390668168000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335631832201-0](offset=623753,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:412}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337620617791, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd6790, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334211402085}, this:0x7f92a3bd6790, timestamp:1727334211402085, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:70, clock:618659840}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277807335692000, v:0}, rec_scn:{val:1726250403931267000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365696606208-2](offset=626688,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170275594572-0](offset=627020,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251879973346-0](offset=627170,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216446493380-0](offset=627396,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a75e40, 0x7f92a3bd6790, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.655543] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=44] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.655548] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.656065] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337620617805, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.656081] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753327496, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753327496, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.656096] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.656101] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.656109] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:444}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753327496, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.656128] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:444}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753327496, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:444}, table_count:1, [{i:0, table_key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8632d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620655392, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=714us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.656152] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C597-0-0, dag_ret:0, dag_status:2, start_time:1727337620652753, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265adb490, output_block_cnt_arr:0x7f9265adb498, concurrent_cnt:1, estimate_row_cnt:4, estimate_occupy_size:6291456, latest_update_ts:1727337620655392, estimated_finish_time:1727337637656154}) [2024-09-26 16:00:20.656159] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d79d00, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C597-0-0, dag_ret:0, dag_status:2, start_time:1727337620652753, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.656164] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=5] task finish process(ret=0, start_time=1727337620655391, end_time=1727337620656164, runtime=773, *this={this:0x7f9264d79d00, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C597-0-0, dag_ret:0, dag_status:2, start_time:1727337620652753, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.656183] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=2] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.656189] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337620617791, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.656197] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753247280, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828753247280, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.656206] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.656208] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.656215] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:412}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753247280, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.656243] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=24] succeed to build merge ctx(tablet_id={id:412}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753247280, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:412}, table_count:1, [{i:0, table_key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620655483, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=730us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.656262] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C596-0-0, dag_ret:0, dag_status:2, start_time:1727337620652301, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a39470, output_block_cnt_arr:0x7f9263a39478, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337620655483, estimated_finish_time:1727337637656265}) [2024-09-26 16:00:20.656268] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d797d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C596-0-0, dag_ret:0, dag_status:2, start_time:1727337620652301, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.656273] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=5] task finish process(ret=0, start_time=1727337620655482, end_time=1727337620656273, runtime=791, *this={this:0x7f9264d797d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C596-0-0, dag_ret:0, dag_status:2, start_time:1727337620652301, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.656290] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=4] schedule one task(task=0x7f926585e3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.656295] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337620617820, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.656303] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752918752, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828752918752, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.656312] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.656315] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.656319] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:309}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752918752, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.656334] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:309}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752918752, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:309}, table_count:1, [{i:0, table_key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}, merge_scn:{val:1726241316899724000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8631da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620655278, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.04ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.656353] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C598-0-0, dag_ret:0, dag_status:2, start_time:1727337620653209, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}, scanned_row_cnt_arr:0x7f9266505490, output_block_cnt_arr:0x7f9266505498, concurrent_cnt:1, estimate_row_cnt:6, estimate_occupy_size:6291456, latest_update_ts:1727337620655278, estimated_finish_time:1727337637656355}) [2024-09-26 16:00:20.656358] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7a230, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C598-0-0, dag_ret:0, dag_status:2, start_time:1727337620653209, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:20.656363] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=5] task finish process(ret=0, start_time=1727337620655276, end_time=1727337620656363, runtime=1087, *this={this:0x7f9264d7a230, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C598-0-0, dag_ret:0, dag_status:2, start_time:1727337620653209, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:20.656394] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=3] schedule one task(task=0x7f926585e750, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.656676] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=444, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5c060, count:6, allocator:0x7f9275dfe7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264d5c390, local_datums:0x7f9264d5c390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:20.656843] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=18][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:20.656881] ERROR issue_dba_error (ob_log.cpp:1875) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.656884] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.656893] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=9][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:20.656911] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=18][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:20.656916] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=5][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92729f2070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92729f20a8, hex: 0300000000000000, int: 3},idx=2:{len: 12, flag: 0, null: 0, ptr: 0x7f92729f21b0, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92729f2118, hex: 420B000000000000, int: 2882},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92729f2150, hex: 288E681FE7280BE8, int: -1726241059036951000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92729f2188, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:24981, column_count:9, rowkey_column_count:6, has_column_checksum:0, row_count:207, row_store_type:0, opt:0, var_column_count:0, row_offset:15610, original_length:16378, max_merged_trans_version:1726241059036951000, data_length:16378, data_zlength:16378, data_checksum:1574137226, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926fa04090, buf_size:16378, data_size:16378, row_count:207, column_count:9, max_merged_trans_version:1726241059036951000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:666643265, row_count_delta:3, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16378}) [2024-09-26 16:00:20.656929] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=13][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:20.656931] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:20.656933] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:20.656934] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:20.656938] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=4][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:20.656940] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:20.656941] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264d5c390, local_datums:0x7f9264d5c390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264d5c390, local_datums:0x7f9264d5c390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f9275dfe118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9275dfe118, limit_param:null, stmt_allocator:0x7f9275dfe7d8, allocator:0x7f9275dfe7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264d5c390, local_datums:0x7f9264d5c390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337620617805, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}) [2024-09-26 16:00:20.657001] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=59][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:20.657002] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264d5c390, local_datums:0x7f9264d5c390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f9275dfe118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9275dfe118, limit_param:null, stmt_allocator:0x7f9275dfe7d8, allocator:0x7f9275dfe7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264d5c390, local_datums:0x7f9264d5c390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337620617805, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}]) [2024-09-26 16:00:20.657039] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=37][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:20.657041] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=1] partition merge iter row count(i=0, row_count=207, ghost_row_count=0, pkey={tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337620617805, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.657053] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:20.657063] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:20.657066] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:20.657068] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337620656634, end_time=1727337620657068, runtime=434, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C597-0-0, dag_ret:0, dag_status:2, start_time:1727337620652753, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.657075] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C597-0-0, dag_ret:0, dag_status:2, start_time:1727337620652753, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:20.657083] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=4330, dag_cnt=36, dag_cnts_[dag.get_type()]=36, &dag=0x7f9264d79940, dag={ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C597-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620652753, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.657091] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C597-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620652748, task_id:YB42C0A8002D-0006230056D8C597-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.657106] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620657104, task_id:YB42C0A8002D-0006230056D8C59B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.657110] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=4] schedule one task(task=0x7f9264d7b1c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.657140] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337620657139, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007354, add_bytes: 0 [2024-09-26 16:00:20.657314] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.657320] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.657325] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.657330] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.657347] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=15] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.657356] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.657415] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=8] multi version scan iterator init succ(param.table_id_=309, range=0x7f92aa6d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:11}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262d98060, count:7, allocator:0x7f92c4a5e7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9262d983b0, local_datums:0x7f9262d983b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:20.657613] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:20.657651] ERROR issue_dba_error (ob_log.cpp:1875) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.657655] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.657663] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:20.657668] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=5][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:20.657670] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:7, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d9c070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 12, flag: 0, null: 0, ptr: 0x7f9262d9c1e8, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d9c0e0, hex: 420B000000000000, int: 2882},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d9c118, hex: 0100000000000000, int: 1, num_digit0: 0},idx=4:{len: 16, flag: 0, null: 0, ptr: 0x7f9262d9c1f4, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d9c188, hex: C818154EB7520BE8, int: -1726195084923955000},idx=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d9c1c0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19292, column_count:11, rowkey_column_count:7, has_column_checksum:0, row_count:156, row_store_type:0, opt:0, var_column_count:0, row_offset:15827, original_length:16391, max_merged_trans_version:1726241316783508000, data_length:16391, data_zlength:16391, data_checksum:3629777942, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6e04090, buf_size:16391, data_size:16391, row_count:156, column_count:11, max_merged_trans_version:1726241316783508000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:619385244, row_count_delta:1, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16391}) [2024-09-26 16:00:20.657694] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=24][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:20.657696] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:20.657698] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:20.657701] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=3][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:20.657703] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:20.657705] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:20.657706] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9262d983b0, local_datums:0x7f9262d983b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d983c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d984d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d985b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d985f0, hex: 9C4DB16500000000, int: 1706118556},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:309}, iter_end:false, schema_rowkey_column_cnt:5, schema_version:1710828752918752, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9262d983b0, local_datums:0x7f9262d983b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d983c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d984d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d985b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d985f0, hex: 9C4DB16500000000, int: 1706118556},]}, store_ctx:{this:0x7f92c4a5e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c4a5e118, limit_param:null, stmt_allocator:0x7f92c4a5e7d8, allocator:0x7f92c4a5e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241316899724000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:590314823027536945, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9262d983b0, local_datums:0x7f9262d983b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d983c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d984d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d985b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d985f0, hex: 9C4DB16500000000, int: 1706118556},]}, key:{"BIGINT":1},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f9279a04108 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x7f927a839198 latest_compact_node=0x7f927aa30240 max_trans_version={val:1726241316783508000, v:0} max_trans_id=125581128 max_elr_trans_version={val:1726241316783508000, v:0} max_elr_trans_id=125581128 latest_compact_ts=1727334209251890 last_compact_cnt=189 total_trans_node_cnt=418 max_modify_scn={val:1726241316783508000, v:0} min_modify_scn={val:1726178872697529000, v:0}}, version_iter:this=0x7f927a8390e8 trans_version={val:1726241316783508000, v:0} scn={val:1726241316783508000, v:0} tx_id={txid:125581128} prev=0x7f927a838f78 next=0x7f927a839198 modify_count=196 acc_checksum=432508559 version=1726211894725631 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:65} seq_no=1726241316536989, multi_version_iter:this=0x7f927aa0b360 trans_version={val:1726194784763860000, v:0} scn={val:1726194784763860000, v:0} tx_id={txid:125216537} prev=0x7f927aa0b2b0 next=0x7f927aa0b9f8 modify_count=107 acc_checksum=2694663391 version=1726178872309143 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no=1726194784741342, max_committed_trans_version:1726241316783508000, cur_trans_version:{val:1726195084923955000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241316899724000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:156, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337620617820, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:11, cur_pos:0, count:0}}) [2024-09-26 16:00:20.657762] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=56][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:20.657763] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:309}, iter_end:false, schema_rowkey_column_cnt:5, schema_version:1710828752918752, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9262d983b0, local_datums:0x7f9262d983b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d983c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d984d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d985b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d985f0, hex: 9C4DB16500000000, int: 1706118556},]}, store_ctx:{this:0x7f92c4a5e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c4a5e118, limit_param:null, stmt_allocator:0x7f92c4a5e7d8, allocator:0x7f92c4a5e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241316899724000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:590314823027536945, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9262d983b0, local_datums:0x7f9262d983b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d983c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d984d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d98510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d985b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9262d985f0, hex: 9C4DB16500000000, int: 1706118556},]}, key:{"BIGINT":1},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f9279a04108 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x7f927a839198 latest_compact_node=0x7f927aa30240 max_trans_version={val:1726241316783508000, v:0} max_trans_id=125581128 max_elr_trans_version={val:1726241316783508000, v:0} max_elr_trans_id=125581128 latest_compact_ts=1727334209251890 last_compact_cnt=189 total_trans_node_cnt=418 max_modify_scn={val:1726241316783508000, v:0} min_modify_scn={val:1726178872697529000, v:0}}, version_iter:this=0x7f927a8390e8 trans_version={val:1726241316783508000, v:0} scn={val:1726241316783508000, v:0} tx_id={txid:125581128} prev=0x7f927a838f78 next=0x7f927a839198 modify_count=196 acc_checksum=432508559 version=1726211894725631 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:65} seq_no=1726241316536989, multi_version_iter:this=0x7f927aa0b360 trans_version={val:1726194784763860000, v:0} scn={val:1726194784763860000, v:0} tx_id={txid:125216537} prev=0x7f927aa0b2b0 next=0x7f927aa0b9f8 modify_count=107 acc_checksum=2694663391 version=1726178872309143 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no=1726194784741342, max_committed_trans_version:1726241316783508000, cur_trans_version:{val:1726195084923955000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241316899724000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:156, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337620617820, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:11, cur_pos:0, count:0}}]) [2024-09-26 16:00:20.657800] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=37][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:20.657802] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=2] partition merge iter row count(i=0, row_count=156, ghost_row_count=0, pkey={tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, table={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337620617820, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.657814] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:20.657823] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=2][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.657827] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620657373, end_time=1727337620657827, runtime=454, *this={this:0x7f926585e750, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C598-0-0, dag_ret:0, dag_status:2, start_time:1727337620653209, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:20.657837] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4628, dag_cnt=35, dag_cnts_[dag.get_type()]=35, &dag=0x7f9264d79e70, dag={ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C598-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620653209, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}) [2024-09-26 16:00:20.657845] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C598-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620653204, task_id:YB42C0A8002D-0006230056D8C598-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.657858] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620657857, task_id:YB42C0A8002D-0006230056D8C59C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.657862] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=4] schedule one task(task=0x7f9264d7b6f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.657913] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=26] multi version scan iterator init succ(param.table_id_=412, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:7}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5c060, count:4, allocator:0x7f92691307d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9264d5c350, local_datums:0x7f9264d5c350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:20.657938] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.657971] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.657974] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.657983] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=0] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337620617791, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.658001] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620657869, end_time=1727337620658001, runtime=132, *this={this:0x7f926585e3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C596-0-0, dag_ret:0, dag_status:2, start_time:1727337620652301, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.658009] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5708, dag_cnt=34, dag_cnts_[dag.get_type()]=34, &dag=0x7f9264d79410, dag={ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C596-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620652301, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.658016] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C596-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620652293, task_id:YB42C0A8002D-0006230056D8C596-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.658029] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620658028, task_id:YB42C0A8002D-0006230056D8C59D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.658033] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=3] schedule one task(task=0x7f9264d7bc20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.658058] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.658061] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.658064] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.658067] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.658071] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8627b60, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:523, tablet_addr:[816017-1849-0-2305843665271992320-2](offset=2043904,size=611,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:458}, data_tablet_id:{id:458}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2897668175, row_count:77572}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86282a8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844621975949312-0](offset=2035712,size=1502,type=2,seq=0), type:"MAJOR", tablet_id:{id:458}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389055265000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844927992370654-0](offset=2037214,size=1787,type=2,seq=0), type:"MINOR", tablet_id:{id:458}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178190631204000, upper_trans_version:1726178190631204000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337620617849, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd1b40, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209496526}, this:0x7f92a3bd1b40, timestamp:1727334209496526, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:61, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274225933800000, v:0}, rec_scn:{val:1726241806895572002, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771761152-2](offset=2039808,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046811469-0](offset=2040141,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881386491-0](offset=2040315,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447906525-0](offset=2040541,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7a490, 0x7f92a3bd1b40, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.658114] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=43] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.658121] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.658696] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337620617864, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.658712] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753417144, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828753417144, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.658791] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=13] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:20.658795] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.658801] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:487}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753417144, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.658822] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:487}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753417144, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:487}, table_count:1, [{i:0, table_key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620657308, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.49ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.658909] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=20] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:20.659027] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59B-0-0, dag_ret:0, dag_status:2, start_time:1727337620657110, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685154a0, output_block_cnt_arr:0x7f92685154a8, concurrent_cnt:1, estimate_row_cnt:2199, estimate_occupy_size:0, latest_update_ts:1727337620657308, estimated_finish_time:1727337637659026}) [2024-09-26 16:00:20.659036] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264d7b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59B-0-0, dag_ret:0, dag_status:2, start_time:1727337620657110, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.659042] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=6] task finish process(ret=0, start_time=1727337620657306, end_time=1727337620659042, runtime=1736, *this={this:0x7f9264d7b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59B-0-0, dag_ret:0, dag_status:2, start_time:1727337620657110, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.659061] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=1] schedule one task(task=0x7f926585e3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.659067] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337620617849, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.659077] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753357664, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753357664, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.659134] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:20.659137] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.659142] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:458}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753357664, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.659160] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:458}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753357664, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:458}, table_count:1, [{i:0, table_key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8627b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620658053, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.09ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.659257] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=18] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:20.659394] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59A-0-0, dag_ret:0, dag_status:2, start_time:1727337620655272, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265adb4a0, output_block_cnt_arr:0x7f9265adb4a8, concurrent_cnt:1, estimate_row_cnt:14598, estimate_occupy_size:4194304, latest_update_ts:1727337620658053, estimated_finish_time:1727337637659395}) [2024-09-26 16:00:20.659401] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7ac90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59A-0-0, dag_ret:0, dag_status:2, start_time:1727337620655272, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.659406] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=5] task finish process(ret=0, start_time=1727337620658051, end_time=1727337620659406, runtime=1355, *this={this:0x7f9264d7ac90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59A-0-0, dag_ret:0, dag_status:2, start_time:1727337620655272, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.659426] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=4] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.659473] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=7][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.659481] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.659484] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.659492] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=5] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.659496] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.659505] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.659526] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.659529] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.659531] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.659535] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.659537] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.659542] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.659595] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=487, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:11}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a66060, count:8, allocator:0x7f92636207d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9264a663e0, local_datums:0x7f9264a663e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:20.659822] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.659866] ERROR issue_dba_error (ob_log.cpp:1875) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.659857] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=458, range=0x7f92aa952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268562060, count:8, allocator:0x7f92c18207d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92685623e0, local_datums:0x7f92685623e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:20.659870] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.659879] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=0] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337620617864, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.659903] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620659555, end_time=1727337620659903, runtime=348, *this={this:0x7f926585e3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59B-0-0, dag_ret:0, dag_status:2, start_time:1727337620657110, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.659916] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2805, dag_cnt=33, dag_cnts_[dag.get_type()]=33, &dag=0x7f9264d7ae00, dag={ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620657110, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.659924] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620657104, task_id:YB42C0A8002D-0006230056D8C59B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.659946] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620659944, task_id:YB42C0A8002D-0006230056D8C59E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.659950] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=3] schedule one task(task=0x7f9264d7c150, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.659955] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337620617893, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.659971] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.659965] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753437040, storage_schema:{this:0x7f9262c13330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=57, max_schema_version_in_memtable=1710828753437040, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.659974] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.659977] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.659981] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.659985] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.659992] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.660026] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1362, total_rows=1362) [2024-09-26 16:00:20.660031] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.660040] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9262c13330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:492}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:12, rowkey_column_count:11, schema_rowkey_col_cnt:9, full_stored_col_cnt:59, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753437040, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=28 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.660072] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=26] succeed to build merge ctx(tablet_id={id:492}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753437040, storage_schema:{this:0x7f9262c13330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:492}, table_count:1, [{i:0, table_key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8624c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620659523, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=515us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.660121] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.660159] ERROR issue_dba_error (ob_log.cpp:1875) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.660160] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=22] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59D-0-0, dag_ret:0, dag_status:2, start_time:1727337620658032, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9262c134c0, output_block_cnt_arr:0x7f9262c134c8, concurrent_cnt:1, estimate_row_cnt:1362, estimate_occupy_size:2097152, latest_update_ts:1727337620659523, estimated_finish_time:1727337637660160}) [2024-09-26 16:00:20.660165] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.660168] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7bc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59D-0-0, dag_ret:0, dag_status:2, start_time:1727337620658032, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.660173] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=5] task finish process(ret=0, start_time=1727337620659522, end_time=1727337620660173, runtime=651, *this={this:0x7f9264d7bc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59D-0-0, dag_ret:0, dag_status:2, start_time:1727337620658032, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.660176] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=0] partition merge iter row count(i=0, row_count=247, ghost_row_count=0, pkey={tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337620617849, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.660193] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337620617879, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.660199] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620659809, end_time=1727337620660199, runtime=390, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59A-0-0, dag_ret:0, dag_status:2, start_time:1727337620655272, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.660202] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753332736, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=36, max_schema_version_in_memtable=1710828753332736, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.660210] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4938, dag_cnt=32, dag_cnts_[dag.get_type()]=32, &dag=0x7f9264d7a8d0, dag={ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620655272, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.660218] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59A-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620655265, task_id:YB42C0A8002D-0006230056D8C59A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.660231] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=2] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.660239] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337620660238, task_id:YB42C0A8002D-0006230056D8C59F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.660243] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=3] schedule one task(task=0x7f9264d7c680, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.660360] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:20.660365] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.660371] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:455}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:10, rowkey_column_count:9, schema_rowkey_col_cnt:7, full_stored_col_cnt:38, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753332736, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=26 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.660415] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:455}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753332736, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:455}, table_count:1, [{i:0, table_key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620659467, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=902us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.660587] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=31] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:20.660667] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337620617907, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.660678] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753441792, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828753441792, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.660736] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1295, total_rows=1295) [2024-09-26 16:00:20.660740] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.660738] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59C-0-0, dag_ret:0, dag_status:2, start_time:1727337620657862, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db94b0, output_block_cnt_arr:0x7f9287db94b8, concurrent_cnt:1, estimate_row_cnt:3824, estimate_occupy_size:4194304, latest_update_ts:1727337620659467, estimated_finish_time:1727337637660739}) [2024-09-26 16:00:20.660746] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d7b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59C-0-0, dag_ret:0, dag_status:2, start_time:1727337620657862, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.660752] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=6] task finish process(ret=0, start_time=1727337620659464, end_time=1727337620660752, runtime=1288, *this={this:0x7f9264d7b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59C-0-0, dag_ret:0, dag_status:2, start_time:1727337620657862, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.660746] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:501}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753441792, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.660774] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:501}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753441792, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:501}, table_count:1, [{i:0, table_key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620659965, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=779us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.660862] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59E-0-0, dag_ret:0, dag_status:2, start_time:1727337620659950, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a39490, output_block_cnt_arr:0x7f9263a39498, concurrent_cnt:1, estimate_row_cnt:1295, estimate_occupy_size:0, latest_update_ts:1727337620659965, estimated_finish_time:1727337637660862}) [2024-09-26 16:00:20.660870] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d7c150, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59E-0-0, dag_ret:0, dag_status:2, start_time:1727337620659950, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.660875] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=5] task finish process(ret=0, start_time=1727337620659963, end_time=1727337620660875, runtime=912, *this={this:0x7f9264d7c150, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59E-0-0, dag_ret:0, dag_status:2, start_time:1727337620659950, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.660891] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=1] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.660898] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=3] schedule one task(task=0x7f9264d852e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.660991] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=492, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:57, schema_rowkey_cnt:9, rowkey_cnt:11, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:9, column_cnt:59}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262daa060, count:11, allocator:0x7f92a33547d8, capacity:11, init_cnt:11}, datum_utils:{is_oracle_mode:false, rowkey_cnt:11, col_cnt:11, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:59, datum_buffer:{capacity:59, datums:0x7f926859a050, local_datums:0x7f9262daa440}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,col_id=43:NOP,col_id=44:NOP,col_id=45:NOP,col_id=46:NOP,col_id=47:NOP,col_id=48:NOP,col_id=49:NOP,col_id=50:NOP,col_id=51:NOP,col_id=52:NOP,col_id=53:NOP,col_id=54:NOP,col_id=55:NOP,col_id=56:NOP,col_id=57:NOP,col_id=58:NOP,]}) [2024-09-26 16:00:20.661197] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=21][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.661235] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.661238] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.661249] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=0] partition merge iter row count(i=0, row_count=61, ghost_row_count=0, pkey={tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337620617893, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.661269] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620660947, end_time=1727337620661269, runtime=322, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59D-0-0, dag_ret:0, dag_status:2, start_time:1727337620658032, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.661279] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3247, dag_cnt=31, dag_cnts_[dag.get_type()]=31, &dag=0x7f9264d7b860, dag={ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620658032, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.661287] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59D-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620658028, task_id:YB42C0A8002D-0006230056D8C59D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.661301] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620661300, task_id:YB42C0A8002D-0006230056D8C5A0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.661306] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=4] schedule one task(task=0x7f9264d7cbb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.661330] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.661334] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.661337] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.661341] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.661345] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b862a0a0, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:490, tablet_addr:[816017-1849-0-2305843629838340096-2](offset=1871872,size=578,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:472}, data_tablet_id:{id:472}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:19976489, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862a734, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844179594129408-0](offset=1847296,size=1090,type=2,seq=0), type:"MAJOR", tablet_id:{id:472}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389523010000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843910084932674-0](offset=1848386,size=839,type=2,seq=0), type:"MINOR", tablet_id:{id:472}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178194887823000, upper_trans_version:1726178194887823000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337620617922, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd4080, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209505593}, this:0x7f92a3bd4080, timestamp:1727334209505593, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:65, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274254072414000, v:0}, rec_scn:{val:1726241829658678000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697839104-2](offset=1859584,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834376524-0](offset=1859916,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881206236-0](offset=1860060,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447726270-0](offset=1860286,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7ac60, 0x7f92a3bd4080, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.661407] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.661416] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.661477] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=501, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268434060, count:6, allocator:0x7f92691307d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9268434390, local_datums:0x7f9268434390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:20.661554] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.661589] ERROR issue_dba_error (ob_log.cpp:1875) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.661593] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.661602] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=0] partition merge iter row count(i=0, row_count=56, ghost_row_count=0, pkey={tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337620617907, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.661622] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620661436, end_time=1727337620661622, runtime=186, *this={this:0x7f9264d852e0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59E-0-0, dag_ret:0, dag_status:2, start_time:1727337620659950, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.661632] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1682, dag_cnt=30, dag_cnts_[dag.get_type()]=30, &dag=0x7f9264d7bd90, dag={ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620659950, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.661640] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59E-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620659944, task_id:YB42C0A8002D-0006230056D8C59E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.661654] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337620661652, task_id:YB42C0A8002D-0006230056D8C5A1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.661659] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=5] schedule one task(task=0x7f9264d7d0e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.661710] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=455, range=0x7f92aac56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:36, schema_rowkey_cnt:7, rowkey_cnt:9, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:7, column_cnt:38}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d36060, count:9, allocator:0x7f92636127d8, capacity:9, init_cnt:9}, datum_utils:{is_oracle_mode:false, rowkey_cnt:9, col_cnt:9, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:38, datum_buffer:{capacity:38, datums:0x7f9287d37340, local_datums:0x7f9287d363f0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,]}) [2024-09-26 16:00:20.661971] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=18][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.662004] ERROR issue_dba_error (ob_log.cpp:1875) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.662007] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.662016] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=0] partition merge iter row count(i=0, row_count=140, ghost_row_count=0, pkey={tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337620617879, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.662034] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620661663, end_time=1727337620662034, runtime=371, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59C-0-0, dag_ret:0, dag_status:2, start_time:1727337620657862, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.662044] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4182, dag_cnt=29, dag_cnts_[dag.get_type()]=29, &dag=0x7f9264d7b330, dag={ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620657862, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.662051] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59C-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620657857, task_id:YB42C0A8002D-0006230056D8C59C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.662081] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620662080, task_id:YB42C0A8002D-0006230056D8C5A2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.662083] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.662085] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=3] schedule one task(task=0x7f9264d7d610, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.662086] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.662088] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726077706339704027) [2024-09-26 16:00:20.662103] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=12] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.662107] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726077706339704027, tablet={this:0x7f92b864a0a0, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[720905-1289-0-2305843650239553536-1](offset=1990656,size=597,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:342}, data_tablet_id:{id:342}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725843677180998000, v:0}, clog_checkpoint_scn:{val:1725843677180998000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726077706339704027, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2338495768, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864a7a0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[720905-1289-0-2305844556477644800-0](offset=1982464,size=1441,type=2,seq=0), type:"MAJOR", tablet_id:{id:342}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725843393342764001}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337620617936, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[720905-1289-0-2305843251881332736-1](offset=1986560,size=226,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[720905-1289-0-2305843183161856226-0](offset=1986786,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[720905-1289-0-2305843251881333124-0](offset=1986948,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[720905-1289-0-2305843216447853158-0](offset=1987174,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7cba0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:20.662140] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=33] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.662146] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.662168] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.662173] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.662175] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.662177] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.662180] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.662184] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.662200] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.662204] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.662205] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.662208] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.662210] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.662214] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.663496] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337620617936, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.663518] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=21] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753001832, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=21, max_schema_version_in_memtable=1710828753001832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.663552] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=29] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.663557] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.663564] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:342}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:23, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753001832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.663585] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:342}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753001832, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:342}, table_count:1, [{i:0, table_key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620662078, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.48ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.663613] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=22] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A0-0-0, dag_ret:0, dag_status:2, start_time:1727337620661305, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a39490, output_block_cnt_arr:0x7f9263a39498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337620662078, estimated_finish_time:1727337637663612}) [2024-09-26 16:00:20.663621] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A0-0-0, dag_ret:0, dag_status:2, start_time:1727337620661305, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.663626] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=5] task finish process(ret=0, start_time=1727337620662076, end_time=1727337620663626, runtime=1550, *this={this:0x7f9264d7cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A0-0-0, dag_ret:0, dag_status:2, start_time:1727337620661305, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.663646] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=2] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.663823] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=342, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:21, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:23}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268594060, count:6, allocator:0x7f92636127d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:23, datum_buffer:{capacity:32, datums:0x7f9268594390, local_datums:0x7f9268594390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,]}) [2024-09-26 16:00:20.663873] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=20][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.663913] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.663917] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.663928] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337620617936, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.663950] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620663777, end_time=1727337620663950, runtime=173, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A0-0-0, dag_ret:0, dag_status:2, start_time:1727337620661305, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.663961] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2656, dag_cnt=28, dag_cnts_[dag.get_type()]=28, &dag=0x7f9264d7c7f0, dag={ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A0-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620661305, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.663969] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A0-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620661300, task_id:YB42C0A8002D-0006230056D8C5A0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.663985] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337620663983, task_id:YB42C0A8002D-0006230056D8C5A3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.663988] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=3] schedule one task(task=0x7f9264d7db40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.664009] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.664012] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.664015] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:20.664020] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.664023] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.664030] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.667226] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337620617978, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.667284] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=57] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.667300] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.667304] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.667310] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101010}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:20.667328] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:101010}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101010}, table_count:1, [{i:0, table_key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8642d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620664003, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.30ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.667366] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A3-0-0, dag_ret:0, dag_status:2, start_time:1727337620663988, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a39470, output_block_cnt_arr:0x7f9263a39478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337620664003, estimated_finish_time:1727337637667366}) [2024-09-26 16:00:20.667375] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d7db40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A3-0-0, dag_ret:0, dag_status:2, start_time:1727337620663988, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.667381] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=5] task finish process(ret=0, start_time=1727337620664000, end_time=1727337620667380, runtime=3380, *this={this:0x7f9264d7db40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A3-0-0, dag_ret:0, dag_status:2, start_time:1727337620663988, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.667412] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.667490] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337620617950, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.667510] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=19] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.667526] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.667530] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.667537] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:140}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.667556] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:140}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:140}, table_count:1, [{i:0, table_key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8622d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620662164, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.37ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.667609] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A1-0-0, dag_ret:0, dag_status:2, start_time:1727337620661659, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505460, output_block_cnt_arr:0x7f9266505468, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337620662164, estimated_finish_time:1727337637667608}) [2024-09-26 16:00:20.667620] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=10] succeed to init merge ctx(task={this:0x7f9264d7d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A1-0-0, dag_ret:0, dag_status:2, start_time:1727337620661659, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.667626] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=5] task finish process(ret=0, start_time=1727337620662163, end_time=1727337620667625, runtime=5462, *this={this:0x7f9264d7d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A1-0-0, dag_ret:0, dag_status:2, start_time:1727337620661659, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.667643] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=4] schedule one task(task=0x7f9287c66750, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.667697] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=7] multi version scan iterator init succ(param.table_id_=101010, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5e060, count:4, allocator:0x7f92636207d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264d5e350, local_datums:0x7f9264d5e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:20.667728] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=140, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268566060, count:3, allocator:0x7f92691307d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9268566320, local_datums:0x7f9268566320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:20.667753] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=28][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.667784] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.667797] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.667800] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.667827] ERROR issue_dba_error (ob_log.cpp:1875) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.667811] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337620617978, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.667830] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.667840] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337620617950, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.667845] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620667650, end_time=1727337620667845, runtime=195, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A3-0-0, dag_ret:0, dag_status:2, start_time:1727337620663988, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.667855] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3867, dag_cnt=27, dag_cnts_[dag.get_type()]=27, &dag=0x7f9264d7d780, dag={ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A3-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620663988, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.667864] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A3-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620663983, task_id:YB42C0A8002D-0006230056D8C5A3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.667863] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620667656, end_time=1727337620667863, runtime=207, *this={this:0x7f9287c66750, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A1-0-0, dag_ret:0, dag_status:2, start_time:1727337620661659, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.667900] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=6240, dag_cnt=26, dag_cnts_[dag.get_type()]=26, &dag=0x7f9264d7cd20, dag={ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A1-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620661659, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.667909] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A1-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337620661652, task_id:YB42C0A8002D-0006230056D8C5A1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.667930] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337620667928, task_id:YB42C0A8002D-0006230056D8C5A4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.667933] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=3] schedule one task(task=0x7f9264d7e070, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.667940] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620667939, task_id:YB42C0A8002D-0006230056D8C5A5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.667942] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=2] schedule one task(task=0x7f9264d7e5a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.668358] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.668367] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=8][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.668370] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:20.668375] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.668379] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.668388] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.668426] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337620617964, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.668439] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9263a3b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.668484] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=40] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.668487] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.668493] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a3b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101009}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:20.668511] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:101009}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9263a3b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101009}, table_count:1, [{i:0, table_key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8641da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620662197, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=6.29ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.668536] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A2-0-0, dag_ret:0, dag_status:2, start_time:1727337620662084, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a3b470, output_block_cnt_arr:0x7f9263a3b478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337620662197, estimated_finish_time:1727337637668535}) [2024-09-26 16:00:20.668542] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7d610, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A2-0-0, dag_ret:0, dag_status:2, start_time:1727337620662084, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.668548] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=5] task finish process(ret=0, start_time=1727337620662196, end_time=1727337620668548, runtime=6352, *this={this:0x7f9264d7d610, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A2-0-0, dag_ret:0, dag_status:2, start_time:1727337620662084, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.668566] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.668569] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337620617922, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.668590] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=20] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753360568, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753360568, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.668599] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.668602] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.668607] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:472}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753360568, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.668624] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:472}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753360568, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:472}, table_count:1, [{i:0, table_key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620661325, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=7.28ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.668641] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59F-0-0, dag_ret:0, dag_status:2, start_time:1727337620660243, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515460, output_block_cnt_arr:0x7f9268515468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337620661325, estimated_finish_time:1727337637668641}) [2024-09-26 16:00:20.668647] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7c680, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59F-0-0, dag_ret:0, dag_status:2, start_time:1727337620660243, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.668653] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=6] task finish process(ret=0, start_time=1727337620661323, end_time=1727337620668653, runtime=7330, *this={this:0x7f9264d7c680, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59F-0-0, dag_ret:0, dag_status:2, start_time:1727337620660243, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.668685] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=3] schedule one task(task=0x7f9262dce030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.668714] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101009, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5e060, count:4, allocator:0x7f92c18227d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264d5e350, local_datums:0x7f9264d5e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:20.668751] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.668788] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.668792] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.668801] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=1] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337620617964, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.668821] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620668679, end_time=1727337620668821, runtime=142, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A2-0-0, dag_ret:0, dag_status:2, start_time:1727337620662084, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.668830] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=6746, dag_cnt=25, dag_cnts_[dag.get_type()]=25, &dag=0x7f9264d7d250, dag={ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A2-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620662084, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.668838] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A2-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620662080, task_id:YB42C0A8002D-0006230056D8C5A2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.668853] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620668851, task_id:YB42C0A8002D-0006230056D8C5A6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.668857] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=3] schedule one task(task=0x7f9264d7ead0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.669360] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337620618007, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.669380] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=20] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=41, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.669395] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.669398] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.669404] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:377}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:43, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.669423] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:377}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:377}, table_count:1, [{i:0, table_key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620668351, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.05ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.669447] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A5-0-0, dag_ret:0, dag_status:2, start_time:1727337620667942, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a39480, output_block_cnt_arr:0x7f9263a39488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337620668351, estimated_finish_time:1727337637669447}) [2024-09-26 16:00:20.669454] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A5-0-0, dag_ret:0, dag_status:2, start_time:1727337620667942, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.669479] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=24] task finish process(ret=0, start_time=1727337620668348, end_time=1727337620669478, runtime=1130, *this={this:0x7f9264d7e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A5-0-0, dag_ret:0, dag_status:2, start_time:1727337620667942, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.669497] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.669556] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=377, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:41, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:43}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268598060, count:5, allocator:0x7f92ba7f67d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:43, datum_buffer:{capacity:43, datums:0x7f92685992b0, local_datums:0x7f9268598360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,]}) [2024-09-26 16:00:20.669599] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.669635] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.669639] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.669649] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=0] partition merge iter row count(i=0, row_count=6, ghost_row_count=0, pkey={tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337620618007, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.669671] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620669510, end_time=1727337620669671, runtime=161, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A5-0-0, dag_ret:0, dag_status:2, start_time:1727337620667942, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.669682] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1740, dag_cnt=24, dag_cnts_[dag.get_type()]=24, &dag=0x7f9264d7e1e0, dag={ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A5-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620667942, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.669689] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A5-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620667939, task_id:YB42C0A8002D-0006230056D8C5A5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.669691] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.669695] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.669698] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:20.669702] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.669711] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620669709, task_id:YB42C0A8002D-0006230056D8C5A7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.669714] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=2] schedule one task(task=0x7f9264d7f000, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.669706] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864eec0, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[814042-5586-0-2305843650238771200-2](offset=1208320,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:297}, data_tablet_id:{id:297}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726063409221604000, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2361333062, row_count:24}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844531780800512-0](offset=1200128,size=1418,type=2,seq=0), type:"MAJOR", tablet_id:{id:297}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063202518238000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843919747962250-0](offset=1201546,size=848,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726063409221604000, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557514739000, upper_trans_version:1726121557514739000}, {i:1, addr:[814042-5586-0-2305843920821704922-0](offset=1202394,size=849,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149603136610000, upper_trans_version:1726149603136610000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337620618021, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd86d0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220478269}, this:0x7f92a3bd86d0, timestamp:1727334220478269, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:74, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726264804632655000, v:0}, rec_scn:{val:1726264804632655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513817088-2](offset=1204224,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843183161074102-0](offset=1204662,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880551000-0](offset=1204824,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447071034-0](offset=1205050,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7eae0, 0x7f92a3bd86d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.669769] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.669777] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.669836] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=472, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a5e060, count:3, allocator:0x7f92691307d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f9264a5e320, local_datums:0x7f9264a5e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:20.669875] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:20.669913] ERROR issue_dba_error (ob_log.cpp:1875) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.669917] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.669923] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:20.669930] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:20.669934] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=3][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e0a8, hex: D01E4FF1C45E0BE8, int: -1726181832211358000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e0e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-27268, column_count:12, rowkey_column_count:3, has_column_checksum:0, row_count:17, row_store_type:0, opt:4, var_column_count:0, row_offset:1040, original_length:1048, max_merged_trans_version:1726238228901957000, data_length:1048, data_zlength:1048, data_checksum:1937654816, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92b7004090, buf_size:1048, data_size:1048, row_count:17, column_count:12, max_merged_trans_version:1726238228901957000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:69434633, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:1048}) [2024-09-26 16:00:20.669947] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:20.669949] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=2][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:00:20.669951] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=2][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:00:20.669953] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=1][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:00:20.669955] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=1][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:00:20.669957] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=1] partition merge iter row count(i=0, row_count=17, ghost_row_count=0, pkey={tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337620617922, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.669985] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=24][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:20.670003] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:20.670009] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=6][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:20.670011] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337620669795, end_time=1727337620670011, runtime=216, *this={this:0x7f9262dce030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59F-0-0, dag_ret:0, dag_status:2, start_time:1727337620660243, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.670018] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262dce030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59F-0-0, dag_ret:0, dag_status:2, start_time:1727337620660243, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:20.670027] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=6] dag finished(dag_ret=-4184, runtime=9784, dag_cnt=23, dag_cnts_[dag.get_type()]=23, &dag=0x7f9264d7c2c0, dag={ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C59F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620660243, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.670035] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C59F-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620660238, task_id:YB42C0A8002D-0006230056D8C59F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.670051] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620670049, task_id:YB42C0A8002D-0006230056D8C5A8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.670055] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=3] schedule one task(task=0x7f9264d7f530, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.670069] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337620618021, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.670075] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.670079] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.670081] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:20.670078] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9263a3d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=16, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.670085] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.670089] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.670090] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.670093] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.670095] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.670098] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a3d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:297}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:18, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.670125] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=23] succeed to build merge ctx(tablet_id={id:297}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9263a3d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:297}, table_count:1, [{i:0, table_key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620669684, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=412us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.670148] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A6-0-0, dag_ret:0, dag_status:2, start_time:1727337620668856, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a3d490, output_block_cnt_arr:0x7f9263a3d498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337620669684, estimated_finish_time:1727337637670148}) [2024-09-26 16:00:20.670154] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7ead0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A6-0-0, dag_ret:0, dag_status:2, start_time:1727337620668856, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.670167] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=12] task finish process(ret=0, start_time=1727337620669682, end_time=1727337620670167, runtime=485, *this={this:0x7f9264d7ead0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A6-0-0, dag_ret:0, dag_status:2, start_time:1727337620668856, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.670190] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.670718] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337620618050, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.670721] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=28] multi version scan iterator init succ(param.table_id_=297, range=0x7f92ab0d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:16, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:18}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5c060, count:6, allocator:0x7f92c244a7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:18, datum_buffer:{capacity:32, datums:0x7f9264d5c390, local_datums:0x7f9264d5c390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,]}) [2024-09-26 16:00:20.670730] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.670750] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.670753] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.670753] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.670758] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:333}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.670789] ERROR issue_dba_error (ob_log.cpp:1875) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.670778] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:333}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:333}, table_count:1, [{i:0, table_key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8651da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620670069, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=687us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.670792] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.670801] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337620618021, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.670808] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A8-0-0, dag_ret:0, dag_status:2, start_time:1727337620670055, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265adb490, output_block_cnt_arr:0x7f9265adb498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337620670069, estimated_finish_time:1727337637670808}) [2024-09-26 16:00:20.670814] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7f530, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A8-0-0, dag_ret:0, dag_status:2, start_time:1727337620670055, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.670819] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=5] task finish process(ret=0, start_time=1727337620670067, end_time=1727337620670819, runtime=752, *this={this:0x7f9264d7f530, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A8-0-0, dag_ret:0, dag_status:2, start_time:1727337620670055, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.670821] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620670677, end_time=1727337620670821, runtime=144, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A6-0-0, dag_ret:0, dag_status:2, start_time:1727337620668856, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.670830] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1974, dag_cnt=22, dag_cnts_[dag.get_type()]=22, &dag=0x7f9264d7e710, dag={ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A6-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620668856, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.670838] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A6-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620668851, task_id:YB42C0A8002D-0006230056D8C5A6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.670863] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=4] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.670871] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620670869, task_id:YB42C0A8002D-0006230056D8C5A9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.670874] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=3] schedule one task(task=0x7f9264d7fa60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.670885] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.670889] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.670892] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:20.670897] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.670901] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.670909] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.671269] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337620618036, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.671283] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.671297] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.671300] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.671306] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101065}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.671324] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:101065}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101065}, table_count:1, [{i:0, table_key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620670879, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=425us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.671347] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A7-0-0, dag_ret:0, dag_status:2, start_time:1727337620669713, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db9490, output_block_cnt_arr:0x7f9287db9498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337620670879, estimated_finish_time:1727337637671346}) [2024-09-26 16:00:20.671353] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7f000, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A7-0-0, dag_ret:0, dag_status:2, start_time:1727337620669713, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.671358] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=5] task finish process(ret=0, start_time=1727337620670877, end_time=1727337620671358, runtime=481, *this={this:0x7f9264d7f000, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A7-0-0, dag_ret:0, dag_status:2, start_time:1727337620669713, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.671376] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=2] schedule one task(task=0x7f926585e3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.671641] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.671646] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.671648] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:20.671652] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.671656] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.671663] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.671846] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101065, range=0x7f92a98d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d30060, count:6, allocator:0x7f9275dfe7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9287d30390, local_datums:0x7f9287d30390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:20.671920] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.671957] ERROR issue_dba_error (ob_log.cpp:1875) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.671961] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.671980] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=0] partition merge iter row count(i=0, row_count=15, ghost_row_count=0, pkey={tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337620618036, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.672004] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620671809, end_time=1727337620672004, runtime=195, *this={this:0x7f926585e3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A7-0-0, dag_ret:0, dag_status:2, start_time:1727337620669713, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.672017] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2304, dag_cnt=21, dag_cnts_[dag.get_type()]=21, &dag=0x7f9264d7ec40, dag={ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A7-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620669713, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.672025] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A7-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620669709, task_id:YB42C0A8002D-0006230056D8C5A7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.672039] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620672038, task_id:YB42C0A8002D-0006230056D8C5AA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.672043] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=4] schedule one task(task=0x7f9264d7ff90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.672066] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.672071] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.672076] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:20.672080] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.672084] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.672091] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.672152] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=333, range=0x7f92aaed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d38060, count:7, allocator:0x7f92c244a7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9287d383b0, local_datums:0x7f9287d383b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:20.672324] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.672357] ERROR issue_dba_error (ob_log.cpp:1875) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.672360] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.672372] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=0] partition merge iter row count(i=0, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337620618050, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.672392] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620672107, end_time=1727337620672392, runtime=285, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A8-0-0, dag_ret:0, dag_status:2, start_time:1727337620670055, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.672402] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=1] dag finished(dag_ret=-4184, runtime=2347, dag_cnt=20, dag_cnts_[dag.get_type()]=20, &dag=0x7f9264d7f170, dag={ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A8-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620670055, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.672409] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A8-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620670049, task_id:YB42C0A8002D-0006230056D8C5A8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.672423] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620672421, task_id:YB42C0A8002D-0006230056D8C5AB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.672427] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=3] schedule one task(task=0x7f9264d804c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.672453] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337620617993, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.672462] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752949376, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=20, max_schema_version_in_memtable=1710828752949376, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.672482] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.672486] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.672491] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:323}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:22, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752949376, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.672509] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:323}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752949376, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:323}, table_count:1, [{i:0, table_key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620671635, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=855us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.672540] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A4-0-0, dag_ret:0, dag_status:2, start_time:1727337620667933, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b21480, output_block_cnt_arr:0x7f9264b21488, concurrent_cnt:1, estimate_row_cnt:205, estimate_occupy_size:0, latest_update_ts:1727337620671635, estimated_finish_time:1727337637672540}) [2024-09-26 16:00:20.672548] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d7e070, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A4-0-0, dag_ret:0, dag_status:2, start_time:1727337620667933, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.672553] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=5] task finish process(ret=0, start_time=1727337620671633, end_time=1727337620672553, runtime=920, *this={this:0x7f9264d7e070, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A4-0-0, dag_ret:0, dag_status:2, start_time:1727337620667933, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.672590] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=2] schedule one task(task=0x7f9287c66750, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.672598] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337620618064, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.672608] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.672619] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.672622] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.672627] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101066}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.672643] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101066}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101066}, table_count:1, [{i:0, table_key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8653ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620672052, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=573us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.672668] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A9-0-0, dag_ret:0, dag_status:2, start_time:1727337620670874, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b23490, output_block_cnt_arr:0x7f9264b23498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337620672052, estimated_finish_time:1727337637672668}) [2024-09-26 16:00:20.672674] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7fa60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A9-0-0, dag_ret:0, dag_status:2, start_time:1727337620670874, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.672679] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=5] task finish process(ret=0, start_time=1727337620672050, end_time=1727337620672679, runtime=629, *this={this:0x7f9264d7fa60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A9-0-0, dag_ret:0, dag_status:2, start_time:1727337620670874, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.672696] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=3] schedule one task(task=0x7f9287c66ae0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.677670] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.677682] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=11][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.677686] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:20.677693] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.677699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706339704027, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8654c80, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:522, tablet_addr:[809012-1445-0-2305843664196722688-2](offset=516096,size=610,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:334}, data_tablet_id:{id:334}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706339704027, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2773161616, row_count:13383}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86553c8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844791625629696-0](offset=507904,size=1660,type=2,seq=0), type:"MAJOR", tablet_id:{id:334}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843977729328764-0](offset=509564,size=902,type=2,seq=0), type:"MINI", tablet_id:{id:334}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337620618078, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bddcc0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221095878}, this:0x7f92a3bddcc0, timestamp:1727334221095878, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:85, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265286142368001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843367843975168-2](offset=512000,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843196045283662-0](offset=512334,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879858684-0](offset=512508,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843215372636894-0](offset=512734,size=192,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc57f0, 0x7f92a3bddcc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.677754] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.677765] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.677795] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.677800] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.677802] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:20.677805] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.677808] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.677812] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.677869] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=323, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:20, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:22}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:5, allocator:0x7f92691327d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,]}) [2024-09-26 16:00:20.678123] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=20][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.678173] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.678177] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.678190] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=0] partition merge iter row count(i=0, row_count=171, ghost_row_count=0, pkey={tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337620617993, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.678218] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620677821, end_time=1727337620678218, runtime=397, *this={this:0x7f9287c66750, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A4-0-0, dag_ret:0, dag_status:2, start_time:1727337620667933, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.678233] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=10299, dag_cnt=19, dag_cnts_[dag.get_type()]=19, &dag=0x7f9264d7dcb0, dag={ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A4-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620667933, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.678242] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A4-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620667928, task_id:YB42C0A8002D-0006230056D8C5A4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.678266] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620678261, task_id:YB42C0A8002D-0006230056D8C5AC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.678275] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=8] schedule one task(task=0x7f9264d80a70, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.678316] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=101066, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:7}, cols_desc:{ObIArray:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:7, allocator:0x7f92bc90a7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9263b5e3b0, local_datums:0x7f9263b5e3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:20.678479] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.678515] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.678519] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.678529] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=0] partition merge iter row count(i=0, row_count=182, ghost_row_count=0, pkey={tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337620618064, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.678550] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620678274, end_time=1727337620678550, runtime=276, *this={this:0x7f9287c66ae0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A9-0-0, dag_ret:0, dag_status:2, start_time:1727337620670874, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.678561] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=7687, dag_cnt=18, dag_cnts_[dag.get_type()]=18, &dag=0x7f9264d7f6a0, dag={ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5A9-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620670874, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.678569] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5A9-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620670869, task_id:YB42C0A8002D-0006230056D8C5A9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.678584] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337620678582, task_id:YB42C0A8002D-0006230056D8C5AD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.678588] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=4] schedule one task(task=0x7f9264d80fa0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.678682] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.678686] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.678691] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=5][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:20.678696] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.678699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864cf80, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[814042-5586-0-2305843644870074368-2](offset=1220608,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:378}, data_tablet_id:{id:378}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2166189977, row_count:350}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864d65c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844352465928192-0](offset=1212416,size=1251,type=2,seq=0), type:"MAJOR", tablet_id:{id:378}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563080818000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843914379265251-0](offset=1213667,size=843,type=2,seq=0), type:"MINI", tablet_id:{id:378}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557779816000, upper_trans_version:1726121557779816000}, {i:1, addr:[814042-5586-0-2305843914379266094-0](offset=1214510,size=843,type=2,seq=0), type:"MINI", tablet_id:{id:378}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986451466000, upper_trans_version:1726149986451466000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337620618107, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdf430, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221743022}, this:0x7f92a3bdf430, timestamp:1727334221743022, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:88, clock:641728512}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293334630000, v:0}, rec_scn:{val:1726265293334630000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513829376-2](offset=1216512,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843176718635446-0](offset=1216950,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880563282-0](offset=1217106,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447083316-0](offset=1217332,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7f2b0, 0x7f92a3bdf430, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.678745] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=46] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.678752] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.678776] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.678778] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.678780] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:20.678783] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.678785] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.678790] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241548806527000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.678812] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337620618092, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.678823] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=14, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.678838] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.678842] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.678848] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:451}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:16, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.678867] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:451}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:451}, table_count:1, [{i:0, table_key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8650e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620677791, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.05ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.678896] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=22] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AB-0-0, dag_ret:0, dag_status:2, start_time:1727337620672426, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db9470, output_block_cnt_arr:0x7f9287db9478, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337620677791, estimated_finish_time:1727337637678896}) [2024-09-26 16:00:20.678903] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d804c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AB-0-0, dag_ret:0, dag_status:2, start_time:1727337620672426, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.678909] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=6] task finish process(ret=0, start_time=1727337620677790, end_time=1727337620678909, runtime=1119, *this={this:0x7f9264d804c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AB-0-0, dag_ret:0, dag_status:2, start_time:1727337620672426, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.678929] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=3] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.678929] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337620618078, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.678947] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=17] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=19, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.679002] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=18] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.679006] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.679012] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:334}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:21, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.679031] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:334}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:334}, table_count:1, [{i:0, table_key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8654c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620677661, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.35ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.679092] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AA-0-0, dag_ret:0, dag_status:2, start_time:1727337620672043, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db74a0, output_block_cnt_arr:0x7f9287db74a8, concurrent_cnt:1, estimate_row_cnt:993, estimate_occupy_size:0, latest_update_ts:1727337620677661, estimated_finish_time:1727337637679084}) [2024-09-26 16:00:20.679099] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7ff90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AA-0-0, dag_ret:0, dag_status:2, start_time:1727337620672043, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.679105] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=5] task finish process(ret=0, start_time=1727337620677657, end_time=1727337620679105, runtime=1448, *this={this:0x7f9264d7ff90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AA-0-0, dag_ret:0, dag_status:2, start_time:1727337620672043, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.679122] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=3] schedule one task(task=0x7f9264ae8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.679167] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=451, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:14, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:16}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:4, allocator:0x7f92636147d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:16, datum_buffer:{capacity:32, datums:0x7f9263b5e350, local_datums:0x7f9263b5e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,]}) [2024-09-26 16:00:20.679193] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.679231] ERROR issue_dba_error (ob_log.cpp:1875) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.679234] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.679243] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337620618092, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.679262] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620679125, end_time=1727337620679261, runtime=136, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AB-0-0, dag_ret:0, dag_status:2, start_time:1727337620672426, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.679271] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=6844, dag_cnt=17, dag_cnts_[dag.get_type()]=17, &dag=0x7f9264d80100, dag={ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AB-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620672426, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.679278] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AB-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620672421, task_id:YB42C0A8002D-0006230056D8C5AB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.679291] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337620679290, task_id:YB42C0A8002D-0006230056D8C5AE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.679295] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=3] schedule one task(task=0x7f9264d814d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.679342] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=334, range=0x7f92ab056200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:19, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:21}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268434060, count:8, allocator:0x7f92636147d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:21, datum_buffer:{capacity:32, datums:0x7f92684343e0, local_datums:0x7f92684343e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,]}) [2024-09-26 16:00:20.679555] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.679607] ERROR issue_dba_error (ob_log.cpp:1875) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.679610] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.679620] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=0] partition merge iter row count(i=0, row_count=153, ghost_row_count=0, pkey={tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337620618078, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.679641] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620679305, end_time=1727337620679641, runtime=336, *this={this:0x7f9264ae8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AA-0-0, dag_ret:0, dag_status:2, start_time:1727337620672043, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.679650] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=7607, dag_cnt=16, dag_cnts_[dag.get_type()]=16, &dag=0x7f9264d7fbd0, dag={ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AA-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620672043, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.679657] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AA-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620672038, task_id:YB42C0A8002D-0006230056D8C5AA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.679671] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620679669, task_id:YB42C0A8002D-0006230056D8C5AF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.679674] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=3] schedule one task(task=0x7f9264d81a00, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.679689] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.679692] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.679695] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:20.679699] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.679703] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b861fe60, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:491, tablet_addr:[816017-1849-0-2305843630912028672-2](offset=1818624,size=579,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:215}, data_tablet_id:{id:215}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3095927781, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86204f4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844118390796288-0](offset=1798144,size=1033,type=2,seq=0), type:"MAJOR", tablet_id:{id:215}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389038459000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843910084883465-0](offset=1799177,size=839,type=2,seq=0), type:"MINOR", tablet_id:{id:215}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726209400113641000, upper_trans_version:1726209400113641000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337620618138, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcf430, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209475373}, this:0x7f92a3bcf430, timestamp:1727334209475373, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:56, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277800266460000, v:0}, rec_scn:{val:1726241800098150000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697789952-2](offset=1810432,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834327372-0](offset=1810764,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881157084-0](offset=1810908,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447677118-0](offset=1811134,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc86d0, 0x7f92a3bcf430, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.679743] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=40] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.679750] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.679765] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:20.679772] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.679842] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.679845] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.679847] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:20.679851] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.679854] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.679860] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.679877] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752511928, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.679890] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.679893] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.679899] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:121}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752511928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.679919] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:121}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620678772, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.12ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.679957] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=31] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AD-0-0, dag_ret:0, dag_status:2, start_time:1727337620678588, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f9265adb480, output_block_cnt_arr:0x7f9265adb488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337620678772, estimated_finish_time:1727337637679956}) [2024-09-26 16:00:20.679972] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=15] succeed to init merge ctx(task={this:0x7f9264d80fa0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AD-0-0, dag_ret:0, dag_status:2, start_time:1727337620678588, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:20.679979] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=6] task finish process(ret=0, start_time=1727337620678771, end_time=1727337620679978, runtime=1207, *this={this:0x7f9264d80fa0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AD-0-0, dag_ret:0, dag_status:2, start_time:1727337620678588, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:20.679997] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=2] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.680000] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337620618138, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.680021] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=20] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752725472, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752725472, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.680033] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.680037] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.680042] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:215}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752725472, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.680057] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:215}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752725472, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:215}, table_count:1, [{i:0, table_key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620679684, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=356us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.680074] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d81110, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AE-0-0, dag_ret:0, dag_status:2, start_time:1727337620679294, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a39460, output_block_cnt_arr:0x7f9263a39468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337620679684, estimated_finish_time:1727337637680074}) [2024-09-26 16:00:20.680080] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d814d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81110, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AE-0-0, dag_ret:0, dag_status:2, start_time:1727337620679294, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.680085] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=4] task finish process(ret=0, start_time=1727337620679683, end_time=1727337620680085, runtime=402, *this={this:0x7f9264d814d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81110, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AE-0-0, dag_ret:0, dag_status:2, start_time:1727337620679294, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.680102] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=3] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.680141] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=121, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5c060, count:5, allocator:0x7f926913c7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9264d5c360, local_datums:0x7f9264d5c360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:20.680158] INFO [STORAGE] dump_info (ob_sstable_struct.cpp:262) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=14] dump merge info(msg="macro block builder close", output_row_per_s=0, new_macro_KB_per_s=0, *this={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:41, merge_start_time:1727337620680117, merge_finish_time:1727337620680158, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:0, max:0, sum:0, count:0}}, {type:"cost_time", info:{min:0, max:0, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:0, max:0, sum:0, count:0}}, {type:"incremental_row_count", info:{min:0, max:0, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:0, snapshot_version:0, start_scn:0, end_scn:0}, macro_id_list:"", comment:""}) [2024-09-26 16:00:20.680174] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=15] partition merge iter row count(i=0, row_count=0, ghost_row_count=0, pkey={tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, table={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337620618123, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.680185] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1544) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=9] merge macro blocks ok(idx_=0, task={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AD-0-0, dag_ret:0, dag_status:2, start_time:1727337620678588, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:20.680194] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=5] task finish process(ret=0, start_time=1727337620680105, end_time=1727337620680194, runtime=89, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AD-0-0, dag_ret:0, dag_status:2, start_time:1727337620678588, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:20.680212] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=3] schedule one task(task=0x7f9265a04250, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.680262] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=215, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:6}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262df8060, count:3, allocator:0x7f92636147d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9262df8320, local_datums:0x7f9262df8320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:20.680288] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.680334] ERROR issue_dba_error (ob_log.cpp:1875) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.680338] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.680348] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=0] partition merge iter row count(i=0, row_count=9, ghost_row_count=0, pkey={tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337620618138, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.680366] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620680215, end_time=1727337620680365, runtime=150, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d81110, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AE-0-0, dag_ret:0, dag_status:2, start_time:1727337620679294, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.680374] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1080, dag_cnt=15, dag_cnts_[dag.get_type()]=15, &dag=0x7f9264d81110, dag={ObIDag:{this:0x7f9264d81110, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AE-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620679294, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.680382] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AE-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620679290, task_id:YB42C0A8002D-0006230056D8C5AE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.680396] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620680393, task_id:YB42C0A8002D-0006230056D8C5B0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.680399] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=3] schedule one task(task=0x7f9264d81f30, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.680406] INFO [STORAGE.COMPACTION] get_merged_sstable (ob_tablet_merge_task.cpp:1285) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=5] create new merged sstable(ctx.param_.tablet_id_={id:121}, snapshot_version=1726241132764173999, ctx.param_.merge_type_=3, ctx.create_snapshot_version_=0, table_mode_flag=0) [2024-09-26 16:00:20.680414] INFO [STORAGE] close (ob_index_block_builder.cpp:815) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=4] succeed to close sstable index builder(res={root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_block_ids_.count():0, other_block_ids_.count():0, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, data_column_cnt:10, data_column_checksums:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], row_count:0, max_merged_trans_version:0, contain_uncommitted_row:false, occupy_size:0, original_size:0, data_checksum:0, use_old_macro_block_count:0, compressor_type:1, root_row_store_type:0, nested_offset:0, nested_size:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}) [2024-09-26 16:00:20.680428] INFO [STORAGE] init (ob_sstable.cpp:118) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=8] succeeded to init sstable(ret=0, this={ObITable:{this:0x7f9265ada128, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9265ada128, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:null, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:20.680449] INFO [STORAGE.COMPACTION] create_sstable (ob_tablet_merge_ctx.cpp:367) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=20] succeed to merge sstable(param={table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_logic_seq:0, schema_version:1710828752511928, create_snapshot_version:0, progressive_merge_round:0, progressive_merge_step:0, is_ready_for_read:true, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, index_type:0, root_block_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), root_block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, root_row_store_type:0, latest_row_store_type:1, data_index_tree_height:0, data_block_macro_meta_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), data_block_macro_meta:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, use_old_macro_block_count:0, row_count:0, rowkey_column_cnt:5, column_cnt:10, column_checksums:[], data_checksum:0, occupy_size:0, original_size:0, max_merged_trans_version:1726241132764173999, ddl_scn:{val:0, v:0}, contain_uncommitted_row:false, is_meta_root:false, compressor_type:1, encrypt_id:0, master_key_id:0, recycle_version:0, nested_offset:0, nested_size:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}, table_key={tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_merge_info={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:1386, merge_start_time:1727337620678772, merge_finish_time:1727337620680158, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"cost_time", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"incremental_row_count", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:1, snapshot_version:0, start_scn:1726211867305630000, end_scn:1726241548806527000}, macro_id_list:"", comment:""}) [2024-09-26 16:00:20.680491] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337620618153, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.680515] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=24] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753355744, storage_schema:{this:0x7f92685c1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753355744, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.680527] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.680531] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.680536] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92685c1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:456}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753355744, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.680553] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:456}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753355744, storage_schema:{this:0x7f92685c1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:456}, table_count:1, [{i:0, table_key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8634c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620679837, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=697us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.680576] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d81640, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AF-0-0, dag_ret:0, dag_status:2, start_time:1727337620679674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685c1490, output_block_cnt_arr:0x7f92685c1498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337620679837, estimated_finish_time:1727337637680576}) [2024-09-26 16:00:20.680582] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d81a00, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81640, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AF-0-0, dag_ret:0, dag_status:2, start_time:1727337620679674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.680587] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=4] task finish process(ret=0, start_time=1727337620679836, end_time=1727337620680587, runtime=751, *this={this:0x7f9264d81a00, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81640, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AF-0-0, dag_ret:0, dag_status:2, start_time:1727337620679674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.680589] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=9][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.680596] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.680599] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:20.680604] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.680609] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=2] schedule one task(task=0x7f9262da4030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.680615] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337620618107, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.680625] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.680633] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.680636] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.680641] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:378}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.680608] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8623ce0, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[816017-1849-0-2305843644870885376-2](offset=2031616,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:154}, data_tablet_id:{id:154}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726074677203981000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3786676695, row_count:578}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86243bc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844492053168128-0](offset=2015232,size=1381,type=2,seq=0), type:"MAJOR", tablet_id:{id:154}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726074015354550000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843989541995877-0](offset=2016613,size=913,type=2,seq=0), type:"MINOR", tablet_id:{id:154}, scn_range:{start_scn:{val:1726074677203981000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726205802467242000, upper_trans_version:1726205802467242000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337620618168, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd5020, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209508489}, this:0x7f92a3bd5020, timestamp:1727334209508489, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:67, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277803107894000, v:0}, rec_scn:{val:1726241804194470000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843367845486592-2](offset=2023424,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719442254-0](offset=2023758,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881370090-0](offset=2023914,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447890124-0](offset=2024140,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc6f60, 0x7f92a3bd5020, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.680663] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=54] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.680655] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:378}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:378}, table_count:1, [{i:0, table_key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620678676, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.96ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.680673] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=6] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.680673] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AC-0-0, dag_ret:0, dag_status:2, start_time:1727337620678274, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505480, output_block_cnt_arr:0x7f9266505488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337620678676, estimated_finish_time:1727337637680672}) [2024-09-26 16:00:20.680678] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=4] succeed to init merge ctx(task={this:0x7f9264d80a70, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AC-0-0, dag_ret:0, dag_status:2, start_time:1727337620678274, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.680682] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=4] task finish process(ret=0, start_time=1727337620678674, end_time=1727337620680682, runtime=2008, *this={this:0x7f9264d80a70, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AC-0-0, dag_ret:0, dag_status:2, start_time:1727337620678274, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.680699] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=3] schedule one task(task=0x7f9265a043c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.680738] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=456, range=0x7f92a9952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268562060, count:7, allocator:0x7f92ba7f67d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f92685623b0, local_datums:0x7f92685623b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:20.680806] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.680846] ERROR issue_dba_error (ob_log.cpp:1875) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.680849] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.680860] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=0] partition merge iter row count(i=0, row_count=45, ghost_row_count=0, pkey={tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337620618153, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.680881] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620680691, end_time=1727337620680881, runtime=190, *this={this:0x7f9262da4030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d81640, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AF-0-0, dag_ret:0, dag_status:2, start_time:1727337620679674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.680894] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=1] dag finished(dag_ret=-4184, runtime=1220, dag_cnt=14, dag_cnts_[dag.get_type()]=14, &dag=0x7f9264d81640, dag={ObIDag:{this:0x7f9264d81640, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AF-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620679674, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.680903] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AF-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620679669, task_id:YB42C0A8002D-0006230056D8C5AF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.680949] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620680947, task_id:YB42C0A8002D-0006230056D8C5B1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.680952] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=3] schedule one task(task=0x7f9264d82460, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.680981] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.680985] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.680990] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=5][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:20.680994] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.681008] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.681016] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.681521] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337620618182, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.681536] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=14] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.681551] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.681555] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.681561] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101015}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.681582] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101015}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101015}, table_count:1, [{i:0, table_key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620680976, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=582us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.681608] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d820a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B1-0-0, dag_ret:0, dag_status:2, start_time:1727337620680952, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d490, output_block_cnt_arr:0x7f9264b1d498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337620680976, estimated_finish_time:1727337637681607}) [2024-09-26 16:00:20.681615] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d82460, type:15, status:2, dag:{ObIDag:{this:0x7f9264d820a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B1-0-0, dag_ret:0, dag_status:2, start_time:1727337620680952, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.681620] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=5] task finish process(ret=0, start_time=1727337620680974, end_time=1727337620681620, runtime=646, *this={this:0x7f9264d82460, type:15, status:2, dag:{ObIDag:{this:0x7f9264d820a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B1-0-0, dag_ret:0, dag_status:2, start_time:1727337620680952, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.681642] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=2] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.681681] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337620618168, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.681691] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.681686] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=101015, range=0x7f92aad52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:6, allocator:0x7f92c18227d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9263b5e390, local_datums:0x7f9263b5e390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:20.681701] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.681704] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.681708] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:154}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.681724] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:154}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:154}, table_count:1, [{i:0, table_key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8623ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620680583, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.12ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.681737] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.681746] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d81b70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B0-0-0, dag_ret:0, dag_status:2, start_time:1727337620680399, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db9480, output_block_cnt_arr:0x7f9287db9488, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337620680583, estimated_finish_time:1727337637681746}) [2024-09-26 16:00:20.681751] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d81f30, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81b70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B0-0-0, dag_ret:0, dag_status:2, start_time:1727337620680399, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.681756] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=5] task finish process(ret=0, start_time=1727337620680580, end_time=1727337620681756, runtime=1176, *this={this:0x7f9264d81f30, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81b70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B0-0-0, dag_ret:0, dag_status:2, start_time:1727337620680399, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.681794] ERROR issue_dba_error (ob_log.cpp:1875) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.681798] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.681808] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337620618182, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.681832] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620681655, end_time=1727337620681832, runtime=177, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d820a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B1-0-0, dag_ret:0, dag_status:2, start_time:1727337620680952, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.681843] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=891, dag_cnt=13, dag_cnts_[dag.get_type()]=13, &dag=0x7f9264d820a0, dag={ObIDag:{this:0x7f9264d820a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B1-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620680952, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.681851] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B1-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620680947, task_id:YB42C0A8002D-0006230056D8C5B1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.681872] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=3] schedule one task(task=0x7f9264b46030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.681880] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620681879, task_id:YB42C0A8002D-0006230056D8C5B2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.681890] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=9] schedule one task(task=0x7f9264d82990, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.682110] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=154, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268434060, count:5, allocator:0x7f92c244a7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9268434360, local_datums:0x7f9268434360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:20.682215] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.682220] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.682230] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337620618168, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.682252] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620682060, end_time=1727337620682251, runtime=191, *this={this:0x7f9264b46030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d81b70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B0-0-0, dag_ret:0, dag_status:2, start_time:1727337620680399, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.682262] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1863, dag_cnt=12, dag_cnts_[dag.get_type()]=12, &dag=0x7f9264d81b70, dag={ObIDag:{this:0x7f9264d81b70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B0-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620680399, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.682270] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B0-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620680393, task_id:YB42C0A8002D-0006230056D8C5B0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.682285] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620682284, task_id:YB42C0A8002D-0006230056D8C5B3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.682288] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=3] schedule one task(task=0x7f9264d82ec0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.682314] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.682318] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.682321] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:20.682325] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.682329] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b863eec0, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:499, tablet_addr:[816017-1849-0-2305843639502065664-2](offset=1921024,size=587,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101016}, data_tablet_id:{id:154}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726074677203981000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:398898191, row_count:578}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844419038609408-0](offset=1900544,size=1313,type=2,seq=0), type:"MAJOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726074015354550000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844024975361313-0](offset=1901857,size=946,type=2,seq=0), type:"MINOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:1726074677203981000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726205802467242000, upper_trans_version:1726205802467242000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337620618196, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd5fc0, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209508808}, this:0x7f92a3bd5fc0, timestamp:1727334209508808, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:69, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277803107894000, v:0}, rec_scn:{val:1726241804194470000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771625984-2](offset=1904640,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161774413-0](offset=1904973,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881251311-0](offset=1905135,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc7f00, 0x7f92a3bd5fc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.682368] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=39] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.682374] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.682393] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.682398] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.682400] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:20.682402] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.682405] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.682409] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.682806] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337620618210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.682819] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.682835] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.682839] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.682844] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101063}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.682875] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=25] succeed to build merge ctx(tablet_id={id:101063}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101063}, table_count:1, [{i:0, table_key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620682390, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=452us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.682901] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B3-0-0, dag_ret:0, dag_status:2, start_time:1727337620682288, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db94a0, output_block_cnt_arr:0x7f9287db94a8, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337620682390, estimated_finish_time:1727337637682900}) [2024-09-26 16:00:20.682908] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d82ec0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B3-0-0, dag_ret:0, dag_status:2, start_time:1727337620682288, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.682914] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=5] task finish process(ret=0, start_time=1727337620682389, end_time=1727337620682913, runtime=524, *this={this:0x7f9264d82ec0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B3-0-0, dag_ret:0, dag_status:2, start_time:1727337620682288, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.682932] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=2] schedule one task(task=0x7f926585e3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.682938] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337620618196, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.682955] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=17] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9266539330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.682967] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.682970] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.682975] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266539330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101016}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.682990] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:101016}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9266539330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101016}, table_count:1, [{i:0, table_key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620682309, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=664us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.683010] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d825d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B2-0-0, dag_ret:0, dag_status:2, start_time:1727337620681890, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266539490, output_block_cnt_arr:0x7f9266539498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337620682309, estimated_finish_time:1727337637683010}) [2024-09-26 16:00:20.683016] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d82990, type:15, status:2, dag:{ObIDag:{this:0x7f9264d825d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B2-0-0, dag_ret:0, dag_status:2, start_time:1727337620681890, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.683023] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=6] task finish process(ret=0, start_time=1727337620682307, end_time=1727337620683022, runtime=715, *this={this:0x7f9264d82990, type:15, status:2, dag:{ObIDag:{this:0x7f9264d825d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B2-0-0, dag_ret:0, dag_status:2, start_time:1727337620681890, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.683053] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=4] schedule one task(task=0x7f9265a04750, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.683091] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=101063, range=0x7f92a9d52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d62060, count:8, allocator:0x7f92ba7f67d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264d623e0, local_datums:0x7f9264d623e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:20.683116] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.683151] ERROR issue_dba_error (ob_log.cpp:1875) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.683154] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.683163] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=0] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337620618210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.683182] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620683055, end_time=1727337620683182, runtime=127, *this={this:0x7f926585e3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B3-0-0, dag_ret:0, dag_status:2, start_time:1727337620682288, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.683191] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=902, dag_cnt=11, dag_cnts_[dag.get_type()]=11, &dag=0x7f9264d82b00, dag={ObIDag:{this:0x7f9264d82b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B3-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620682288, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.683198] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B3-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620682284, task_id:YB42C0A8002D-0006230056D8C5B3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.683211] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620683210, task_id:YB42C0A8002D-0006230056D8C5B4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.683214] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=3] schedule one task(task=0x7f9264d833f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.683233] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=16][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.683236] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.683239] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:20.683243] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.683247] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.683253] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.683257] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101016, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f926859a060, count:6, allocator:0x7f92636167d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f926859a390, local_datums:0x7f926859a390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:20.683342] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.683345] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.683353] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337620618196, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.683371] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620683221, end_time=1727337620683371, runtime=150, *this={this:0x7f9265a04750, type:1, status:2, dag:{ObIDag:{this:0x7f9264d825d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B2-0-0, dag_ret:0, dag_status:2, start_time:1727337620681890, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.683379] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1489, dag_cnt=10, dag_cnts_[dag.get_type()]=10, &dag=0x7f9264d825d0, dag={ObIDag:{this:0x7f9264d825d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B2-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620681890, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.683386] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B2-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620681879, task_id:YB42C0A8002D-0006230056D8C5B2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.683398] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620683396, task_id:YB42C0A8002D-0006230056D8C5B5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.683401] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=2] schedule one task(task=0x7f9264d83920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.683457] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.683461] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.683464] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:20.683470] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.683474] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.683502] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=16] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.684083] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337620618239, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.684084] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337620618224, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.684101] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=17] update column cnt(column_cnt_=26, store_column_cnt_=26, column_cnt_=26, column_array_.count()=0) [2024-09-26 16:00:20.684116] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=31] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=15, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.684106] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=4] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9268515480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=26, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.684133] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=12] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.684134] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=23] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.684137] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.684138] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.684143] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:452}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:17, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.684144] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:328}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:28, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.684162] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:452}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9263a39330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:452}, table_count:1, [{i:0, table_key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8652d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620683227, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=913us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.684162] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:328}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9268515480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:328}, table_count:1, [{i:0, table_key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e072bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620683452, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=689us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.684185] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B5-0-0, dag_ret:0, dag_status:2, start_time:1727337620683400, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685155d0, output_block_cnt_arr:0x7f92685155d8, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337620683452, estimated_finish_time:1727337637684185}) [2024-09-26 16:00:20.684186] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B4-0-0, dag_ret:0, dag_status:2, start_time:1727337620683214, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a39480, output_block_cnt_arr:0x7f9263a39488, concurrent_cnt:1, estimate_row_cnt:10, estimate_occupy_size:0, latest_update_ts:1727337620683227, estimated_finish_time:1727337637684185}) [2024-09-26 16:00:20.684192] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d83920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B5-0-0, dag_ret:0, dag_status:2, start_time:1727337620683400, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.684193] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d833f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B4-0-0, dag_ret:0, dag_status:2, start_time:1727337620683214, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.684198] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=5] task finish process(ret=0, start_time=1727337620683225, end_time=1727337620684198, runtime=973, *this={this:0x7f9264d833f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B4-0-0, dag_ret:0, dag_status:2, start_time:1727337620683214, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.684198] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=5] task finish process(ret=0, start_time=1727337620683450, end_time=1727337620684197, runtime=747, *this={this:0x7f9264d83920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B5-0-0, dag_ret:0, dag_status:2, start_time:1727337620683400, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.684241] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.684249] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=5] schedule one task(task=0x7f9287c66750, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.684304] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=328, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:26, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:28}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268594060, count:5, allocator:0x7f92691a87d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:28, datum_buffer:{capacity:32, datums:0x7f9268594360, local_datums:0x7f9268594360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,]}) [2024-09-26 16:00:20.684389] ERROR issue_dba_error (ob_log.cpp:1875) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.684393] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.684403] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337620618239, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.684426] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620684260, end_time=1727337620684426, runtime=166, *this={this:0x7f9287c66750, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B5-0-0, dag_ret:0, dag_status:2, start_time:1727337620683400, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.684435] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1035, dag_cnt=9, dag_cnts_[dag.get_type()]=9, &dag=0x7f9264d83560, dag={ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B5-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620683400, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.684443] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B5-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620683396, task_id:YB42C0A8002D-0006230056D8C5B5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.684466] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337620684464, task_id:YB42C0A8002D-0006230056D8C5B6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.684469] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=3] schedule one task(task=0x7f9264d83e50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.684528] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=452, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:15, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:17}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5e060, count:5, allocator:0x7f92636227d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:17, datum_buffer:{capacity:32, datums:0x7f9264d5e360, local_datums:0x7f9264d5e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,]}) [2024-09-26 16:00:20.684604] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.684608] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.684618] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=0] partition merge iter row count(i=0, row_count=10, ghost_row_count=0, pkey={tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337620618224, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.684637] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620684466, end_time=1727337620684637, runtime=171, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B4-0-0, dag_ret:0, dag_status:2, start_time:1727337620683214, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.684647] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1433, dag_cnt=8, dag_cnts_[dag.get_type()]=8, &dag=0x7f9264d83030, dag={ObIDag:{this:0x7f9264d83030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B4-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620683214, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.684655] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B4-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620683210, task_id:YB42C0A8002D-0006230056D8C5B4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.684670] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620684668, task_id:YB42C0A8002D-0006230056D8C5B7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.684674] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=3] schedule one task(task=0x7f9264d84380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.684703] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.684708] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.684711] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:20.684715] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.684719] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.684726] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.684747] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.684749] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.684751] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:20.684754] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.684757] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.684761] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.685154] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337620618267, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.685168] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=12] update column cnt(column_cnt_=29, store_column_cnt_=29, column_cnt_=29, column_array_.count()=0) [2024-09-26 16:00:20.685171] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9262c13490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.685185] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.685188] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.685193] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9262c13490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:329}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.685212] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:329}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9262c13490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:329}, table_count:1, [{i:0, table_key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e089f20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620684744, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=447us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.685238] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B7-0-0, dag_ret:0, dag_status:2, start_time:1727337620684674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9262c135f0, output_block_cnt_arr:0x7f9262c135f8, concurrent_cnt:1, estimate_row_cnt:62, estimate_occupy_size:0, latest_update_ts:1727337620684744, estimated_finish_time:1727337637685238}) [2024-09-26 16:00:20.685244] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d84380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B7-0-0, dag_ret:0, dag_status:2, start_time:1727337620684674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.685250] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=6] task finish process(ret=0, start_time=1727337620684742, end_time=1727337620685250, runtime=508, *this={this:0x7f9264d84380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B7-0-0, dag_ret:0, dag_status:2, start_time:1727337620684674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.685268] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=2] schedule one task(task=0x7f9262d98030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.685284] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337620618253, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.685294] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=9] update column cnt(column_cnt_=12, store_column_cnt_=12, column_cnt_=12, column_array_.count()=0) [2024-09-26 16:00:20.685297] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9263a39490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.685314] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.685317] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.685322] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a39490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:330}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.685339] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:330}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9263a39490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:330}, table_count:1, [{i:0, table_key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e107fe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620684698, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=622us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.685364] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B6-0-0, dag_ret:0, dag_status:2, start_time:1727337620684469, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a395f0, output_block_cnt_arr:0x7f9263a395f8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337620684698, estimated_finish_time:1727337637685363}) [2024-09-26 16:00:20.685369] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d83e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B6-0-0, dag_ret:0, dag_status:2, start_time:1727337620684469, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.685374] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=5] task finish process(ret=0, start_time=1727337620684696, end_time=1727337620685374, runtime=678, *this={this:0x7f9264d83e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B6-0-0, dag_ret:0, dag_status:2, start_time:1727337620684469, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.685390] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=5] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.685428] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=329, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5e060, count:6, allocator:0x7f92c18227d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9264d5e390, local_datums:0x7f9264d5e390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:20.685603] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.685609] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.685620] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=0] partition merge iter row count(i=0, row_count=47, ghost_row_count=0, pkey={tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337620618267, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.685640] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620685393, end_time=1727337620685640, runtime=247, *this={this:0x7f9262d98030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B7-0-0, dag_ret:0, dag_status:2, start_time:1727337620684674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.685651] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=977, dag_cnt=7, dag_cnts_[dag.get_type()]=7, &dag=0x7f9264d83fc0, dag={ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B7-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620684674, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.685659] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B7-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620684668, task_id:YB42C0A8002D-0006230056D8C5B7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.685673] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620685671, task_id:YB42C0A8002D-0006230056D8C5B8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.685678] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=5] schedule one task(task=0x7f9264d848b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.685698] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.685702] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.685704] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:20.685708] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.685713] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.685709] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=330, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5e060, count:7, allocator:0x7f92691a87d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9264d5e3b0, local_datums:0x7f9264d5e3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:20.685720] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.685895] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.685929] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.685932] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.685942] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=0] partition merge iter row count(i=0, row_count=141, ghost_row_count=0, pkey={tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337620618253, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.685960] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620685683, end_time=1727337620685960, runtime=277, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B6-0-0, dag_ret:0, dag_status:2, start_time:1727337620684469, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.685969] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1500, dag_cnt=6, dag_cnts_[dag.get_type()]=6, &dag=0x7f9264d83a90, dag={ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B6-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620684469, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.685976] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B6-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620684464, task_id:YB42C0A8002D-0006230056D8C5B6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.685989] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337620685987, task_id:YB42C0A8002D-0006230056D8C5B9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.685993] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=4] schedule one task(task=0x7f9264d84de0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.686021] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.686025] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:20.686027] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:20.686032] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.686036] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b86a2d40, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:2, version:2, length:511, tablet_addr:[816017-1849-0-2305843652387110912-2](offset=2064384,size=599,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101067}, data_tablet_id:{id:334}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1854554341, row_count:13383}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86a3488, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844595132416000-0](offset=2048000,size=1477,type=2,seq=0), type:"MAJOR", tablet_id:{id:101067}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843951961064901-0](offset=2049477,size=878,type=2,seq=0), type:"MINI", tablet_id:{id:101067}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}, {i:1, addr:[816017-1849-0-2305843958403516723-0](offset=2050355,size=884,type=2,seq=0), type:"MINI", tablet_id:{id:101067}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726208018709503000, upper_trans_version:1726208018709503000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337620618296, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bde490, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221276956}, this:0x7f92a3bde490, timestamp:1727334221276956, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:86, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265292880560000, v:0}, rec_scn:{val:1726264918599518000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843482735894528-2](offset=2056192,size=441,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046827961-0](offset=2056633,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881402983-0](offset=2056807,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bcbd80, 0x7f92a3bde490, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:20.686079] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=43] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.686085] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.686412] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337620618282, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.686427] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=25, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.686442] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.686446] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.686452] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:332}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:27, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.686474] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:332}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:332}, table_count:1, [{i:0, table_key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8648b00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620685693, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=756us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.686500] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B8-0-0, dag_ret:0, dag_status:2, start_time:1727337620685678, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515490, output_block_cnt_arr:0x7f9268515498, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337620685693, estimated_finish_time:1727337637686500}) [2024-09-26 16:00:20.686507] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d848b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B8-0-0, dag_ret:0, dag_status:2, start_time:1727337620685678, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.686512] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=5] task finish process(ret=0, start_time=1727337620685690, end_time=1727337620686512, runtime=822, *this={this:0x7f9264d848b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B8-0-0, dag_ret:0, dag_status:2, start_time:1727337620685678, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.686530] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=2] schedule one task(task=0x7f9287c66750, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.686865] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=332, range=0x7f92aae56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:25, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:27}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:6, allocator:0x7f92636147d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:27, datum_buffer:{capacity:32, datums:0x7f9263b5e390, local_datums:0x7f9263b5e390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,]}) [2024-09-26 16:00:20.686904] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=19][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.686941] ERROR issue_dba_error (ob_log.cpp:1875) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.686944] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.686954] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337620618282, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.686972] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620686836, end_time=1727337620686972, runtime=136, *this={this:0x7f9287c66750, type:1, status:2, dag:{ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B8-0-0, dag_ret:0, dag_status:2, start_time:1727337620685678, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.686982] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1303, dag_cnt=5, dag_cnts_[dag.get_type()]=5, &dag=0x7f9264d844f0, dag={ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B8-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620685678, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.686989] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B8-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337620685671, task_id:YB42C0A8002D-0006230056D8C5B8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.687036] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337620618296, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:20.687045] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:20.687064] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:20.687067] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:20.687072] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101067}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:20.687091] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101067}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101067}, table_count:1, [{i:0, table_key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b86a2d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337620686016, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.05ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:20.687120] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B9-0-0, dag_ret:0, dag_status:2, start_time:1727337620685993, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db94a0, output_block_cnt_arr:0x7f9287db94a8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337620686016, estimated_finish_time:1727337637687120}) [2024-09-26 16:00:20.687126] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d84de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B9-0-0, dag_ret:0, dag_status:2, start_time:1727337620685993, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.687131] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=5] task finish process(ret=0, start_time=1727337620686014, end_time=1727337620687131, runtime=1117, *this={this:0x7f9264d84de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B9-0-0, dag_ret:0, dag_status:2, start_time:1727337620685993, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.687398] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=3] schedule one task(task=0x7f9265a04750, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:20.687470] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101067, range=0x7f92ab152200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d38060, count:8, allocator:0x7f92ba7f67d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d383e0, local_datums:0x7f9287d383e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:20.687659] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=36][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.687697] ERROR issue_dba_error (ob_log.cpp:1875) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.687700] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.687708] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=0] partition merge iter row count(i=0, row_count=227, ghost_row_count=0, pkey={tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337620618296, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.687731] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620687419, end_time=1727337620687731, runtime=312, *this={this:0x7f9265a04750, type:1, status:2, dag:{ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B9-0-0, dag_ret:0, dag_status:2, start_time:1727337620685993, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.687761] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=18] dag finished(dag_ret=-4184, runtime=1767, dag_cnt=4, dag_cnts_[dag.get_type()]=4, &dag=0x7f9264d84a20, dag={ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5B9-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620685993, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.687770] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5B9-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337620685987, task_id:YB42C0A8002D-0006230056D8C5B9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.688801] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=378, range=0x7f92aa952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268596060, count:5, allocator:0x7f92c18227d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9268596360, local_datums:0x7f9268596360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:20.688869] ERROR issue_dba_error (ob_log.cpp:1875) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.688873] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.688883] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337620618107, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.688906] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620688755, end_time=1727337620688905, runtime=150, *this={this:0x7f9265a043c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AC-0-0, dag_ret:0, dag_status:2, start_time:1727337620678274, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.688916] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=10642, dag_cnt=3, dag_cnts_[dag.get_type()]=3, &dag=0x7f9264d80630, dag={ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AC-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620678274, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.688924] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AC-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620678261, task_id:YB42C0A8002D-0006230056D8C5AC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.689026] INFO [STORAGE] build_new_table_store (ob_tablet_table_store.cpp:1206) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=21] succeed to build new table store(major_tables_={cnt:1, is_inited:true}, minor_tables_={cnt:3, is_inited:true}, memtables_={count:1}, PRINT_TS(*this)={ObTabletTableStore_Pretty:this:0x7f92aa7cfca8, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, memtables:{count:1}, is_ready_for_read:true, table_array:{ [2871] [ YB42C0A8002D-0006230056D8C5AD-0-0 ] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [2871] [ YB42C0A8002D-0006230056D8C5AD-0-0 ] MAJOR MAJOR 1725120003420849000 1725120003420849000 {val:0, v:0} {val:1726077600415496000, v:0} 0 false [2871] [ YB42C0A8002D-0006230056D8C5AD-0-0 ] MINOR MINOR 1726178585269925002 1726178585269925002 {val:1726077259159067000, v:0} {val:1726178585269925002, v:0} 0 false [2871] [ YB42C0A8002D-0006230056D8C5AD-0-0 ] MINI 1726211867106157000 1726211867106157000 {val:1726178585269925002, v:0} {val:1726211867305630000, v:0} 0 false [2871] [ YB42C0A8002D-0006230056D8C5AD-0-0 ] MINI 1726241132764173999 1726241132764173999 {val:1726211867305630000, v:0} {val:1726241548806527000, v:0} 0 false [2871] [ YB42C0A8002D-0006230056D8C5AD-0-0 ] MEM ACTIVE 9223372036854775807 18446744073709547609 {val:1726241548806527000, v:0} {val:4611686018427387903, v:0} 3 unused }}) [2024-09-26 16:00:20.690842] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=16646144, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:20.690873] INFO [COMMON] wash (ob_kvcache_store.cpp:335) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash memory, (tenant_id=1, cache_size=16646144, lower_mem_limit=2147483648, upper_mem_limit=2147483648, min_wash_size=0, max_wash_size=0, mem_usage=1305686016, reserve_mem=137180449, wash_size=2254405632) [2024-09-26 16:00:20.690953] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash time detail, (compute_wash_size_time=50, refresh_score_time=14, wash_time=10) [2024-09-26 16:00:20.691563] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=2] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=1258280, clean_num=31457) [2024-09-26 16:00:20.695227] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337619695413]) [2024-09-26 16:00:20.695241] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337619695413], now0=[mts=1727337619695413], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:20.695249] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337619694790, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:20.695271] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=21][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:20.695282] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337619694790, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:20.695303] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=20] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337619694790, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:20.695328] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=24][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:20.695335] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:20.695339] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:20.695343] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:20.695346] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:20.695351] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=16, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:20.695356] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] will sleep(sleep_us=16000, remain_us=12856472, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:20.697426] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337620278717]}) [2024-09-26 16:00:20.702002] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:20.702016] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337620701998) [2024-09-26 16:00:20.702022] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337620601945, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:20.702054] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=223349937965, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.702061] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:20.702070] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:20.702075] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:20.702078] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:20.702084] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:20.702087] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:20.702089] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337620702037) [2024-09-26 16:00:20.705398] INFO [MDS] init_for_evict_medium_info (ob_tablet_mds_data.cpp:414) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=15] succeeded to init mds data(ret=0, finish_medium_scn=1726077600415496000, src_medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, src_extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}) [2024-09-26 16:00:20.705437] INFO [STORAGE] init_for_merge (ob_tablet.cpp:363) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=18] succeeded to init tablet for mini/minor/major merge(ret=0, param={sstable:0x7f9265ada128, snapshot_version:1726241132764173999, clog_checkpoint_scn:{val:1726241548806527000, v:0}, multi_version_start:1726178585269925002, need_report:false, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, rebuild_seq:0, update_with_major_flag:false, need_check_sstable:false, ddl_info:{keep_old_ddl_sstable:true, ddl_start_scn:{val:0, v:0}, ddl_commit_scn:{val:0, v:0}, ddl_checkpoint_scn:{val:0, v:0}, ddl_snapshot_version:0, ddl_execution_id:-1, data_format_version:0}, allow_duplicate_sstable:false, merge_type:"MINI_MERGE", need_check_transfer_seq:true, transfer_seq:0}, old_tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337620616659000, hold_ref_cnt:true, ref_cnt:3, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337620618123, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}, this={this:0x7f9287db6060, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x7f92aa7d0450, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726241548806527000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726241132764173999, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f9287db6670, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}, {i:2, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726241132764173999, upper_trans_version:1726241132764173999}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x7f9287db6520, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9287db7880, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9287db7950, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{redefined:false, snapshot_version:0, schema_version:0, data_tablet_id:{id:0}, hidden_tablet_id:{id:0}, lob_meta_tablet_id:{id:50121}, lob_piece_tablet_id:{id:60121}, is_old_mds:true}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:20.705579] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=122] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287db8060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}}, ref_cnt:0, upper_trans_version:1725120003420849000, timestamp:0}, this:0x7f9287db8060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1725120003420849000, max_merged_trans_version:1725120003420849000, data_macro_block_count:1, nested_size:4096, nested_offset:589824, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:84, row_count_:14, occupy_size_:1604, original_size_:1914, data_checksum_:3587602083, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:1, data_micro_block_count_:1, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:2, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1725120003420849000, max_merged_trans_version_:1725120003420849000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287db8310, column_checksum_count:10, data_root_info:{addr:[0-0-0-3458764933653594112-0](offset=0,size=391,type=3,seq=0), block_data:{buf:0x7f9287db8360, size:391, extra_buf:0x7f9287db84e7, extra_size:424, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-3458765009889263616-0](offset=0,size=462,type=3,seq=0), block_data:{buf:0x7f9287db868f, size:462, extra_buf:null, extra_size:0, type:1}}, data_block_count:1, other_block_count:0, linked_block_count:0, data_block_ids:0x7f9287db885d, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:true, nested_offset:589824, nested_size:4096}}, valid_for_reading:true}) [2024-09-26 16:00:20.705604] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=23] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287db8060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}}, ref_cnt:0, upper_trans_version:1726178585269925002, timestamp:0}, this:0x7f9287db8060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726178585269925002, max_merged_trans_version:1726178585269925002, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726178585269925002, v:0}, meta:{basic_meta:{version:1, length:94, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726178585269925002, max_merged_trans_version_:1726178585269925002, recycle_version:1726077600415496000, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726178585269925002, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:1, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287db8310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:20.705616] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=11] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287db8060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}}, ref_cnt:0, upper_trans_version:1726211867106157000, timestamp:0}, this:0x7f9287db8060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726211867106157000, max_merged_trans_version:1726211867106157000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726211867305630000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726211867106157000, max_merged_trans_version_:1726211867106157000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726211867305630000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287db8310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:20.705626] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=9] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287db8060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9287db8060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287db8310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:20.705637] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=9][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.705690] ERROR issue_dba_error (ob_log.cpp:1875) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.705695] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf9de5b1 0xf9dcfd6 0xe51c6fb 0xe519c03 0xe518f89 0xe516f8c 0xf01158d 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.705716] INFO [STORAGE] reset (ob_tablet.cpp:159) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=0] reset tablet(this=0x7f9287db6060, ls_id={id:1}, tablet_id={id:121}, lbt()="0x121f1f9c 0xe52cc09 0xe52c6c5 0xe52bd13 0x4fd348f 0xf01110c 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:20.705726] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1250) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=0][errcode=-4184] sstable merge finish(ret=-4184, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AD-0-0, dag_ret:0, dag_status:2, start_time:1727337620678588, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f9265adb480, output_block_cnt_arr:0x7f9265adb488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337620678772, estimated_finish_time:1727337637679956}, compaction_filter:NULL, time_guard:total=1.70ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, task={this:0x7f9265a04250, type:16, status:2, dag:{ObIDag:{this:0x7f9264d80be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AD-0-0, dag_ret:0, dag_status:2, start_time:1727337620678588, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:20.705763] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620680405, end_time=1727337620705762, runtime=25357, *this={this:0x7f9265a04250, type:16, status:2, dag:{ObIDag:{this:0x7f9264d80be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AD-0-0, dag_ret:0, dag_status:2, start_time:1727337620678588, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:20.705775] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=27187, dag_cnt=2, dag_cnts_[dag.get_type()]=2, &dag=0x7f9264d80be0, dag={ObIDag:{this:0x7f9264d80be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5AD-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620678588, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}) [2024-09-26 16:00:20.705785] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5AD-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620678582, task_id:YB42C0A8002D-0006230056D8C5AD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.710279] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:20.710289] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.711836] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:16, local_retry_times:16, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:20.711849] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:20.711853] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:20.711855] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:20.711861] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:20.711901] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:20.711906] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:20.719492] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:20.727510] INFO [STORAGE] write_meta_block (ob_index_block_builder.cpp:1808) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=3] succeed to write macro meta in macro block(ret=0, macro_meta_={val:{version:1, length:156, data_checksum:2073194008, rowkey_count:4, rowkey_count:4, column_count:4, micro_block_count:144, occupy_size:2055851, data_size:7595160, data_zsize:2046419, original_size:7595160, progressive_merge_round:6, block_offset:2055851, block_size:7155, row_count:124307, row_count_delta:-47188, max_merged_trans_version:1726190662817616000, is_encrypted:false, is_deleted:false, contain_uncommitted_row:false, compressor_type:2, master_key_id:0, encrypt_id:0, encrypt_key:"", row_store_type:0, schema_version:1711680543204904, snapshot_version:1726241132764173999, is_last_row_last_flag:false, logic_id:{data_seq:0, logic_version:1726241290462324999, tablet_id:1152921504606847223}, macro_id:[1812](ver=0,mode=0,seq=824627), column_checksums:[0, 0, 0, 0], has_string_out_row:false, all_lob_in_row:false}, end_key:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f926859e140, hex: 3230323430393133303232303238, cstr: 20240913022028},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f926859e14e, hex: 343132323136313031, cstr: 412216101},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f926859e0e0, hex: 686266CBA26D0BE8, int: -1726165486201839000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f926859e118, hex: 0000000000000000, int: 0},]store_rowkey:}}) [2024-09-26 16:00:20.727608] INFO [STORAGE] write_meta_block (ob_index_block_builder.cpp:1808) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=5] succeed to write macro meta in macro block(ret=0, macro_meta_={val:{version:1, length:156, data_checksum:1746792960, rowkey_count:4, rowkey_count:4, column_count:4, micro_block_count:144, occupy_size:2053874, data_size:8207786, data_zsize:2044442, original_size:8207786, progressive_merge_round:6, block_offset:2053874, block_size:7310, row_count:134087, row_count_delta:-47820, max_merged_trans_version:1726190662817616000, is_encrypted:false, is_deleted:false, contain_uncommitted_row:false, compressor_type:2, master_key_id:0, encrypt_id:0, encrypt_key:"", row_store_type:0, schema_version:1711680543204904, snapshot_version:1726241132764173999, is_last_row_last_flag:false, logic_id:{data_seq:0, logic_version:1726241290462324999, tablet_id:1152921504606847222}, macro_id:[1091](ver=0,mode=0,seq=824626), column_checksums:[0, 0, 0, 0], has_string_out_row:false, all_lob_in_row:false}, end_key:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f926856c140, hex: 3230323430393133303232323034, cstr: 20240913022204},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f926856c14e, hex: 343132333337363234, cstr: 412337624},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f926856c0e0, hex: 50DCFA50B36D0BE8, int: -1726165415241262000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f926856c118, hex: 0000000000000000, int: 0},]store_rowkey:}}) [2024-09-26 16:00:20.727864] INFO [STORAGE.BLKMGR] async_write (ob_macro_block_handle.cpp:178) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=24] Async write macro block(macro_id_=[1812](ver=0,mode=0,seq=824627)) [2024-09-26 16:00:20.727876] INFO [STORAGE] flush (ob_macro_block.cpp:864) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=10] macro block writer succeed to flush macro block.(block_id=[1812](ver=0,mode=0,seq=824627), common_header_={header_size:24, version:1, magic:1001, attr:1, payload_size:2063243, payload_checksum:-359285756}, macro_header_={fixed_header:{header_size:192, version:2, magic:1007, tablet_id:1152921504606847223, logical_version:1726241290462324999, data_seq:0, column_count:4, rowkey_column_count:4, row_store_type:0, row_count:124307, occupy_size:2055851, micro_block_count:144, micro_block_data_offset:216, micro_block_data_size:2055635, idx_block_offset:2055851, idx_block_size:7155, meta_block_offset:2063006, meta_block_size:261, data_checksum:2073194008, compressor_type:2, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", col_type_array_cnt:4}, column_types:0x7f92b6a040e8, column_orders:0x7f92b6a040f8, column_checksum:0x7f92b6a04108, column_checksum:[0, 0, 0, 0]}, contain_uncommitted_row=false, max_merged_trans_version=1726190662817616000, ¯o_handle=0x7f92664efe28) [2024-09-26 16:00:20.727896] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.727977] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.727983] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1bc0 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.728016] INFO [STORAGE.BLKMGR] async_write (ob_macro_block_handle.cpp:178) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=30] Async write macro block(macro_id_=[1091](ver=0,mode=0,seq=824626)) [2024-09-26 16:00:20.727999] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=0] partition merge iter row count(i=0, row_count=4508, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.728024] INFO [STORAGE] flush (ob_macro_block.cpp:864) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=7] macro block writer succeed to flush macro block.(block_id=[1091](ver=0,mode=0,seq=824626), common_header_={header_size:24, version:1, magic:1001, attr:1, payload_size:2061421, payload_checksum:45797750}, macro_header_={fixed_header:{header_size:192, version:2, magic:1007, tablet_id:1152921504606847222, logical_version:1726241290462324999, data_seq:0, column_count:4, rowkey_column_count:4, row_store_type:0, row_count:134087, occupy_size:2053874, micro_block_count:144, micro_block_data_offset:216, micro_block_data_size:2053658, idx_block_offset:2053874, idx_block_size:7310, meta_block_offset:2061184, meta_block_size:261, data_checksum:1746792960, compressor_type:2, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", col_type_array_cnt:4}, column_types:0x7f926fe040e8, column_orders:0x7f926fe040f8, column_checksum:0x7f926fe04108, column_checksum:[0, 0, 0, 0]}, contain_uncommitted_row=false, max_merged_trans_version=1726190662817616000, ¯o_handle=0x7f92685e7e28) [2024-09-26 16:00:20.728037] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:20.728032] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=27] partition merge iter row count(i=1, row_count=119763, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337620617654, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.728071] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337620637027, end_time=1727337620728071, runtime=91044, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C590-0-0, dag_ret:0, dag_status:2, start_time:1727337620618622, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.728095] ERROR issue_dba_error (ob_log.cpp:1875) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:20.728095] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=109473, dag_cnt=1, dag_cnts_[dag.get_type()]=1, &dag=0x7f9264d774f0, dag={ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C590-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620618622, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.728100] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1bc0 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:20.728106] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C590-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620618620, task_id:YB42C0A8002D-0006230056D8C590-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.728112] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=0] partition merge iter row count(i=0, row_count=4249, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.728134] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=18] partition merge iter row count(i=1, row_count=130147, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337620617639, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:20.728215] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337620637036, end_time=1727337620728214, runtime=91178, *this={this:0x7f9287c663c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58F-0-0, dag_ret:0, dag_status:2, start_time:1727337620618617, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:20.728227] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=109610, dag_cnt=0, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264d76fc0, dag={ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C58F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337620618617, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:20.728236] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C58F-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337620618613, task_id:YB42C0A8002D-0006230056D8C58F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:20.740556] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:20.740565] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.757260] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757275] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=11] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60476}, multi_version_start=1726077614845725000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077614845725000, old_tablet_snaphsot_version=1726077689155434000, tablet={this:0x7f92b8655c20, is_inited:true, wash_score:-7496034416099556807, hold_ref_cnt:true, ref_cnt:2, version:2, length:476, tablet_addr:[674007-5161-0-2305843614805131264-0](offset=1048576,size=564,type=2,seq=0), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:60476}, data_tablet_id:{id:476}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726077689155434000, multi_version_start:1726077614845725000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:0, row_count:0}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86562b4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:0}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[674007-5161-0-2305843315231137792-0](offset=1024000,size=285,type=2,seq=0), type:"MAJOR", tablet_id:{id:60476}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077600415496000}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{null}}}, addr:[674007-5161-0-2305843251880390656-0](offset=1044480,size=226,type=2,seq=0)}, storage_schema_addr:{ptr:null, addr:[674007-5161-0-2305843163833561314-0](offset=1044706,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[674007-5161-0-2305843251880391026-0](offset=1044850,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0}) [2024-09-26 16:00:20.757315] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=40] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60476}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757331] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757334] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200426}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757352] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=16] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757355] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60379}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757359] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757361] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50380}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757365] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757368] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200048}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757371] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757374] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847407}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757377] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757380] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200450}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757383] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757385] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200197}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757389] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757392] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:233}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757396] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757398] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200127}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757402] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757404] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50114}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757407] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757410] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50192}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757413] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757415] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200251}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757419] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757422] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60350}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757425] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757428] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60429}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757431] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757433] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60259}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757437] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757439] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847145}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757445] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757447] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:188}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757450] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757464] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=13] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847104}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757468] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757470] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:224}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757473] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757476] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:308}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757481] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757484] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:369}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757488] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757490] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200325}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757493] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757496] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:493}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757499] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757501] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101011}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757505] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757507] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847061}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757511] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757513] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846994}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757516] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757521] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50124}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757526] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757544] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=17] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200124}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757548] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757550] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:284}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757553] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757555] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50149}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757558] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757560] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:307}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757564] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50240}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757569] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757572] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50355}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757575] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757577] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60457}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757583] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757585] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200312}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757589] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757601] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=11] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:109}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757606] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757607] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:182}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757611] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757613] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101098}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757617] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757619] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60387}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757623] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757625] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847037}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757629] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757631] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847198}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757634] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757637] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60297}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757640] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757642] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60305}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757646] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757648] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60342}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757652] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757654] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757658] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757660] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847084}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757663] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757665] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847417}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757671] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757673] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101036}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757676] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757678] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200107}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757681] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757683] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847359}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757686] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757689] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60005}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757693] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757695] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60130}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757698] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757701] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847232}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757704] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757706] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200301}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757710] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757712] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101064}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757718] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757720] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60129}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757724] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757726] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:404}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757730] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757732] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200130}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757735] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757738] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50393}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757741] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757743] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101080}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757747] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757749] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200052}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757753] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757755] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200063}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757761] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757763] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50424}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757766] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757768] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200031}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757771] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757773] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200314}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757776] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757778] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:197}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757781] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757783] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50381}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757786] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757788] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200492}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757792] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757794] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:249}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757797] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757800] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200332}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757805] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757807] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:103}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757811] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757812] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50373}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757815] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757818] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200304}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757821] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757823] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200300}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757826] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757829] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846997}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757832] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757834] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847013}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757838] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757842] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50437}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757846] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757848] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200206}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757851] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757853] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101083}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757859] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757862] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847294}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757865] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757869] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200433}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757873] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757875] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101062}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757878] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757880] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60263}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757883] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757885] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200480}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757891] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757904] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:152}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757908] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757911] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847199}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757914] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757916] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:129}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757922] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757924] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50364}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757927] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757929] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200364}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757932] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757935] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50279}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757939] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757941] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101018}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757945] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757947] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60167}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757951] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757953] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60424}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757957] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757959] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60438}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757962] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757965] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60440}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757968] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757970] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847055}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757974] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757976] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101034}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757982] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757984] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200262}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757987] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757989] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847006}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757993] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.757995] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:255}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.757998] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758000] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101084}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758004] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758006] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50219}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758010] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758012] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60233}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758017] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758025] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=8] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200086}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758029] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758031] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50142}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758034] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758036] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200163}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758039] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758041] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50120}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758045] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758047] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50406}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758053] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=5] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758056] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50315}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758059] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758061] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200183}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758064] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758078] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=13] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847002}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758082] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758084] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101068}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758088] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758090] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60140}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758094] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847081}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758100] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758102] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200136}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758105] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758107] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:439}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758111] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758113] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50484}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758116] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758119] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200109}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758122] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758124] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:165}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758130] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758132] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:157}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758135] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758137] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60145}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758140] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758142] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847155}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758148] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758150] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:363}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758153] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758155] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847345}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758159] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758161] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200310}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758164] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758167] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60444}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758170] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758172] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50329}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758176] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758178] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60226}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758181] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758183] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:225}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758187] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758189] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200330}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758193] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758195] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60441}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758198] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758200] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847357}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758206] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758208] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847046}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758211] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758224] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=12] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200448}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758229] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758231] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847099}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758237] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758239] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847363}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758242] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758244] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60253}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758248] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758250] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101073}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758253] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758256] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200178}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758259] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758261] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847349}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758264] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758267] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60383}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758271] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758273] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:128}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758276] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758279] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200034}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758282] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758285] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:427}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758290] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758292] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:294}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758295] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758297] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758301] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758303] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847242}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758306] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758309] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200122}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758312] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758315] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:392}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758318] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758320] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50379}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758324] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758326] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50450}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758329] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200493}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758337] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758339] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:196}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758342] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758344] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60243}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758347] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758349] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200184}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758355] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758357] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847362}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758360] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758362] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:349}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758365] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758367] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60166}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758371] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758373] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847228}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758377] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758379] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847289}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758383] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758385] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847394}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758388] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758391] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60484}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758394] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758397] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200168}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758400] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758402] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847378}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758406] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758408] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50334}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758412] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758414] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101031}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758417] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758420] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60459}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758423] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758426] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846983}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758429] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758431] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60425}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758434] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758436] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50408}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758440] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758442] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60349}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758445] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758447] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847418}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758452] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758454] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846990}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758458] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758460] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847049}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758463] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758465] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60113}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758469] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758471] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847416}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758475] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758477] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60352}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758482] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758485] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847057}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758488] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758491] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847200}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758494] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758498] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60180}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758502] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758504] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:473}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758518] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758521] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50325}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758525] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758527] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847105}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758530] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758532] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200021}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758535] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758537] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847187}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758540] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758542] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60431}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758546] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758548] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50292}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758552] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758554] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60410}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758558] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758560] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:372}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758563] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50140}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758569] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758571] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60491}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758575] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758577] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758580] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758583] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:429}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758586] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758588] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60165}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758594] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758596] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200171}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758599] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758601] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101005}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758604] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758606] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:362}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758609] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758611] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200025}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758614] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758616] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:405}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758619] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758621] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60309}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758624] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758627] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101027}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758632] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758634] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200204}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758638] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758639] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846986}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758643] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758645] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:210}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758648] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758651] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847402}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758654] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758656] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60103}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758660] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758662] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:00:20.758665] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758668] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50246}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758671] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758673] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50251}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758678] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758681] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846977}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758684] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758685] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847101}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758693] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=7] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758695] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847036}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758701] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758703] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200038}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758706] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758708] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847344}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758711] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758714] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60262}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758717] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758719] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847282}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758723] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758725] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50003}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758728] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758731] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200491}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758736] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758738] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60279}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758742] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758743] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60355}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758746] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758748] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60354}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758752] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758754] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60488}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758757] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758760] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:381}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758763] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758765] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60329}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758768] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758771] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101082}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758774] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758776] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:344}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758780] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758782] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200464}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758785] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758788] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60430}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758791] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758793] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50263}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758797] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758799] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50115}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758803] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758805] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60104}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758808] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758810] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60245}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758815] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758817] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50239}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758820] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758822] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:432}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758826] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758828] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847110}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758831] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758833] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758837] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758839] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60478}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758842] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758844] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200346}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758850] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758852] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847157}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758855] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758857] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:327}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758860] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758862] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847281}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758866] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758868] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200215}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758871] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758873] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60248}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758877] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758879] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50109}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758885] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758887] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847364}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758890] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758892] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200419}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758895] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758898] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200009}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758901] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758904] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:410}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758907] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758909] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60147}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758912] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758914] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847076}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758918] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758920] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200337}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758923] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758926] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847292}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758929] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758931] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60361}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758935] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758937] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60116}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758940] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758942] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847043}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758945] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758948] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60323}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758951] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758953] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101057}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758956] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758959] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101001}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758962] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758966] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60158}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758970] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758972] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60358}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758975] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758977] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200101}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758980] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758982] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847321}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758985] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758987] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:423}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758990] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758993] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60334}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.758996] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.758998] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50261}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759004] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759006] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200363}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759009] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759011] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60296}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759014] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759017] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200342}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759020] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759022] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60381}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759026] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759028] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60380}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759031] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759033] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200452}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759037] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759039] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50391}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759052] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759055] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60487}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759059] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759061] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101033}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759065] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759067] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50479}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759070] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759072] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:316}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759075] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759077] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60251}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759080] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759082] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60293}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759085] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759088] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50392}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759093] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759095] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200340}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759099] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759101] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60181}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759104] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759106] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200456}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759112] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759114] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759117] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759119] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50305}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759122] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759124] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847056}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759130] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759132] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50121}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759135] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759137] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:232}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759140] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759142] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:320}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759146] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759148] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846980}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759151] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759153] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847354}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759157] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759159] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200094}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759163] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759167] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50401}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759171] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759173] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:499}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759176] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759179] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60119}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759182] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759185] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50004}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759188] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759190] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50475}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759194] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759214] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=15] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:166}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759226] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=8] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759231] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101030}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759236] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759238] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847111}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759242] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759245] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200247}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759248] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759250] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101094}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759253] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759256] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101020}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759259] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759261] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50306}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759265] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759267] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50117}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759271] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759273] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200451}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759277] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759279] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101044}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759282] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759284] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847406}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759288] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759290] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847272}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759294] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759296] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847346}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759300] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759302] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847217}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759305] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759308] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100002}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759312] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759314] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60452}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759317] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759320] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200200}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759323] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759326] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200460}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759329] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759332] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60386}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759335] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759337] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60133}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759341] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759343] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50167}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759346] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759348] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50440}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759352] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759354] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:292}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759358] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759360] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:293}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759364] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759368] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200040}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759372] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759374] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50395}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759378] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759380] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847396}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759397] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=14] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759402] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:290}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759406] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759408] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:118}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759411] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759413] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50146}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759417] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759419] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50207}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759423] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759425] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200228}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759429] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759431] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:371}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759434] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759436] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847191}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759440] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759442] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200454}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759448] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759450] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847086}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759454] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759469] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=14] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847039}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759474] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759476] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847381}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759480] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759482] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847139}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759486] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759488] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:388}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759492] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759494] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847365}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759498] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759500] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60450}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759503] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759506] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200020}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759509] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759512] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:445}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759515] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759517] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200141}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759523] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759525] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101017}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759528] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759530] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847278}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759534] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759536] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50455}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759540] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759542] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847370}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759545] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759547] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847375}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759551] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759553] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200030}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759556] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759559] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101013}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759562] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759565] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:192}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759570] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759572] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759583] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=9] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759586] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60294}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759592] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759594] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200045}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759597] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759599] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60214}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759602] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759604] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847077}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759608] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759609] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60398}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759613] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759615] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:401}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759618] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759621] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200485}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759624] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759626] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847033}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759630] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759632] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60256}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759636] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759638] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759641] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759644] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101072}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759647] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759650] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50458}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759654] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759656] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60432}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759659] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759661] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50431}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759667] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759669] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101088}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759672] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759674] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50150}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759678] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759680] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101069}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759684] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759686] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847286}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759691] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759693] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847087}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759697] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847015}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759703] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759705] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847420}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759708] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759711] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:244}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759714] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759717] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847270}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759720] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759722] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847361}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759726] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759730] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200296}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759733] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759735] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50357}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759738] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759742] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50245}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759746] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759748] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60493}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759752] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759755] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50410}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759758] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759760] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60351}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759763] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759766] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847154}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759769] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759771] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60106}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759776] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759778] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200066}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759782] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759784] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60402}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759788] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759790] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60108}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759794] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759795] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:191}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759801] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759803] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200209}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759820] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=15] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759822] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847118}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759826] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759828] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847014}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759832] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759834] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50249}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759838] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759840] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200350}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759843] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759845] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50128}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759849] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759851] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101060}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759855] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759857] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200069}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759860] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759863] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200210}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759867] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759869] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50256}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759872] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759875] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:252}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759878] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759882] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200229}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759886] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759888] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:279}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759891] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759894] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200037}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759897] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759899] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:296}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759903] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759905] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200422}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759908] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759911] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60336}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759914] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759916] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:357}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759919] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759921] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50442}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759925] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759927] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101022}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759931] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759933] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:151}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759936] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759938] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50362}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759942] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759945] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50110}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759948] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759950] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200462}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759953] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759955] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200302}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759959] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759961] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50252}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759964] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759967] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50471}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759970] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759972] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:325}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759976] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759978] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50226}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759981] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759983] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200423}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759987] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759989] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50488}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759993] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.759995] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50214}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.759998] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760001] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60412}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760004] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760006] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760010] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760013] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847414}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760016] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760033] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:305}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760039] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760042] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847250}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760046] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760049] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847029}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760053] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760056] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50428}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760060] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760062] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50113}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760066] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760069] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200437}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760073] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760076] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60451}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760080] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760083] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200177}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760087] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760089] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60404}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760094] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760097] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847372}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760100] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760102] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200486}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760104] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760107] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50441}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760110] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760112] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:389}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760116] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760118] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:5}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760122] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760124] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101035}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760128] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760130] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60327}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760133] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760135] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847151}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760139] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760141] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50145}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760144] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760147] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60426}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760150] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760152] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101058}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760157] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760160] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:407}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760163] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760165] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847141}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760168] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760171] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101081}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760174] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760176] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:245}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760180] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760182] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101086}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760185] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760189] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200111}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760193] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760196] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101052}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760199] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760201] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50133}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760204] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760206] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101019}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760210] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760212] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60157}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760216] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760219] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847291}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760223] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760226] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:403}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760229] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760231] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200064}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760237] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760239] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101079}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760242] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760244] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50307}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760247] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760249] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50272}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760253] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760255] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:383}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760259] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760261] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847403}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760264] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760267] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:238}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760270] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760272] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200430}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760276] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760278] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60306}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760281] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760284] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50333}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760287] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760289] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847188}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760293] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760295] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:150}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760298] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760301] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847285}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760304] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760306] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200220}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760309] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760312] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50238}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760315] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760317] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:406}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760321] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760323] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847011}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760326] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760329] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760332] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760335] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847156}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760338] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760341] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60330}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760344] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760347] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50319}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760350] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760352] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50377}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760355] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760358] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847273}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760363] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760957] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:20.760966] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=5] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760969] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:396}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760972] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760974] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200205}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760977] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760979] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60320}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760982] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760984] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200196}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.760987] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.760990] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200494}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761000] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761004] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60407}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761007] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761009] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200487}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761013] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761015] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847293}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761022] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=5] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761024] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50430}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761027] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761029] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200010}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761032] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761035] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50104}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761040] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761042] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50425}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761045] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761047] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50480}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761050] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761052] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200154}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761055] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761058] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60390}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761061] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761063] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50436}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761067] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761069] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50160}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761072] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761074] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200120}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761077] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761079] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:474}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761083] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761085] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60393}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761088] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761090] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847051}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761094] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50451}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761099] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761102] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50366}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761105] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761107] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847279}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761110] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761112] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60148}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761116] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761118] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200137}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761122] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761124] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200006}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761128] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761130] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60472}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761133] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761136] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60004}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761139] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761141] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:391}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761144] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761147] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:426}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761151] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761154] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:348}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761159] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761161] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50385}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761165] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761167] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847180}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761171] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761173] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50438}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761176] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761178] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200311}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761182] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761187] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=5] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200159}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761191] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761193] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847007}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761196] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761197] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200051}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761201] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761202] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:235}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761205] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761207] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200316}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761210] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761212] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50336}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761215] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761219] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847153}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761223] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761225] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200453}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761228] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761231] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60401}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761234] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761236] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:397}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761240] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761242] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:110}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761245] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761247] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200211}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761251] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761253] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200216}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761257] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761259] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60254}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761262] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761264] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:202}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761270] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761272] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200092}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761275] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761277] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60146}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761280] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761282] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:319}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761288] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761290] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50423}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761293] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761295] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:385}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761298] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761301] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60111}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761305] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761308] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50208}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761311] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761313] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200068}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761318] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761320] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200074}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761323] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761325] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847373}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761328] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761330] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50342}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761333] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761334] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50223}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761341] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761343] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60244}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761347] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761349] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60439}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761352] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761355] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200157}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761360] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761362] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50452}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761365] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761367] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:155}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761371] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761373] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50309}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761377] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761378] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200112}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761382] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761384] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60003}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761387] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761389] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200438}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761393] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761395] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50349}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761399] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761401] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847422}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761404] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761406] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50446}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761409] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761412] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:253}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761417] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761419] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50125}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761422] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761424] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:382}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761428] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761430] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846979}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761433] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761436] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847024}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761439] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761442] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847129}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761444] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761447] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200036}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761452] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761456] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60344}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761459] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761461] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50157}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761465] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761466] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:350}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761469] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761471] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200110}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761474] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761476] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847367}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761479] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761483] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847283}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761487] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761489] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60149}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761492] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761494] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761498] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761500] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50189}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761505] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761507] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:291}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761510] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761512] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200165}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761529] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=15] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761532] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50492}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761535] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761538] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60471}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761541] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761544] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200175}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761547] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761549] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761553] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761555] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60332}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761558] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761560] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60405}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761566] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761568] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60150}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761571] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761573] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200128}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761576] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761578] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60213}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761582] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761584] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50188}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761587] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761589] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60375}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761593] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761595] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200018}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761598] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761601] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847121}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761604] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761606] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50403}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761609] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761612] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101043}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761615] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761617] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761621] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761623] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:315}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761626] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761628] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50378}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761631] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761633] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847380}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761637] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761639] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50225}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761642] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761644] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50409}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761649] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761652] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200125}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761655] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761657] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200338}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761661] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761663] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200429}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761666] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761668] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:167}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761671] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761674] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60121}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761678] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761680] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:149}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761684] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761686] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200365}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761689] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761691] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200115}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761694] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761697] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200303}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761700] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761702] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200129}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761705] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761707] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847275}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761713] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761715] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200263}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761718] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761720] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60151}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761723] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761726] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50232}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761729] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761731] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200182}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761735] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761737] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200444}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761741] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761744] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847280}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761747] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761750] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200024}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761753] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761755] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50236}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761759] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761761] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:143}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761764] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761767] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:189}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761769] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761773] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846988}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761776] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761779] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:446}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761782] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761784] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60246}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761788] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761790] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101032}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761796] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761798] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60345}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761801] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761803] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:208}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761807] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761809] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200008}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761812] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761814] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847350}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761817] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761819] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50356}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761822] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761824] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:124}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761827] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761829] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:326}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761832] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761835] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60455}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761838] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761840] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50331}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761843] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761846] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200428}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761849] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761851] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200005}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761854] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761856] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200208}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761859] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761862] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50457}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761865] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761867] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200308}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761871] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761873] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50332}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761876] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761879] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60128}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761882] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761884] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50472}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761887] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761890] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50474}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761893] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761895] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200449}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761898] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761901] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50248}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761904] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761906] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:107}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761909] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761912] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200173}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761915] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761918] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50324}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761921] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761923] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50432}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761926] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761928] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60168}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761932] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761934] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:102}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761937] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761939] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60392}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761944] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761946] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50296}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761950] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761952] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60236}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761955] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761957] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:478}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761961] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761963] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50294}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761966] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761969] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50358}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761980] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761984] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847097}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761989] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761992] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50360}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.761996] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.761999] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762002] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762007] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200001}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762010] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762014] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200118}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762017] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762020] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101028}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762023] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762025] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:376}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762029] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762031] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:239}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762035] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762037] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847054}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762040] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762042] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200139}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762047] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762050] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50253}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762053] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762055] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762058] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762060] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50132}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762063] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762065] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60378}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762068] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762071] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200138}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762074] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762077] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:411}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762081] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762084] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50155}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762087] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762091] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101021}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762094] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60324}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762099] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762101] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50343}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762104] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762106] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50250}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762111] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762113] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847271}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762117] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762119] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847295}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762122] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762124] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200484}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762127] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762130] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200160}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762133] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762135] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50487}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762138] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762140] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60292}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762143] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762145] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60196}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762148] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762150] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50350}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762154] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762156] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60239}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762159] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762161] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200071}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762164] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762176] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=11] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50498}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762181] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762183] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:111}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762187] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762189] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847063}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762192] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762194] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847053}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762197] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762200] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60416}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762203] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762207] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50111}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762211] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762212] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:477}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762215] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762218] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:209}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762223] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762226] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60366}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762229] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762231] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60369}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762234] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762236] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200468}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762239] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762241] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847424}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762245] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762247] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60110}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762251] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762253] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200258}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762256] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762260] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50116}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762264] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762266] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:443}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762269] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762271] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50375}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762275] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762277] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200082}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762280] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762283] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847140}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762286] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762288] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101037}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762293] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762295] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847117}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762299] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762301] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:390}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762305] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762307] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:263}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762310] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762313] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:135}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762316] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762318] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:335}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762323] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762325] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101054}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762329] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60363}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762335] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762337] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200465}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762340] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762342] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50444}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762346] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762348] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:153}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762352] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762355] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847080}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762359] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762361] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200472}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762365] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762367] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50165}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762370] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762373] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847098}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762376] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762378] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847415}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762383] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762392] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=8] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60395}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762396] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762398] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60364}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762401] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762403] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50119}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762406] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762408] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847160}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762414] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762416] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60433}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762419] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762421] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:374}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762424] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762426] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50005}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762431] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762433] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847412}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762436] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762438] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50123}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762442] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762444] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101041}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762447] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762449] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200436}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762453] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762455] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200246}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762461] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762463] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50369}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762466] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762468] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200445}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762471] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762473] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60370}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762475] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762477] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200133}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762480] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762483] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847341}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762486] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762489] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847423}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762492] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762494] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200050}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762497] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762500] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50399}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762503] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762505] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200313}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762511] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762513] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200424}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762517] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762518] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200070}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762521] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762524] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60210}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762527] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762529] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200269}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762532] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762535] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101024}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762538] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762540] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50147}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762544] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762546] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847004}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762549] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762551] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:236}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762557] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762559] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50316}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762562] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762564] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:237}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762567] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762570] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60225}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762573] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762575] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60131}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762578] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762581] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200123}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762584] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762588] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847027}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762591] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762594] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50190}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762597] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762600] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847060}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762603] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762605] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60423}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762609] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762611] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:483}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762614] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762617] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50439}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762621] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762624] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60391}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762627] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762629] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101055}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762632] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762634] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50400}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762637] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762640] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60357}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762643] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762645] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:113}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762650] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762652] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200416}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762656] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762658] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60458}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762661] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762663] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:123}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762667] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762669] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847028}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762672] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762674] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50365}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762680] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762683] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50290}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762697] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762700] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847128}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762703] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762705] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200151}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762708] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762711] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60437}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762714] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762716] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847351}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762722] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762724] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50433}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762727] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762729] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50135}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762732] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762734] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200425}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762737] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762740] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200155}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762743] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762745] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847181}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762748] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762751] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:471}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762756] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762758] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:450}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762762] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762764] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762769] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762771] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50376}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762775] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762777] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200169}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762780] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762782] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60397}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762786] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762788] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:375}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762792] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762794] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847152}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762799] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762802] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:3}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762807] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762809] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60360}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762812] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762814] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60112}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762817] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762818] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:219}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762822] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762824] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60333}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762827] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762831] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:168}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762835] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762838] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200076}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762841] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762843] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846985}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762848] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762851] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:141}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762854] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762855] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847253}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762859] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762861] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200061}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762863] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762866] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:475}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762869] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762871] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60399}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762875] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762877] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101040}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762881] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762883] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:259}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762886] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762889] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847083}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762892] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762894] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846978}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762897] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762899] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50491}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762903] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762905] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762909] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762911] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:248}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762914] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762916] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847038}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762920] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762922] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200176}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762925] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762927] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847089}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762931] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762933] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200043}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762936] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762939] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200078}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762945] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762954] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=8] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847120}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762958] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762959] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60153}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762963] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762965] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50386}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762970] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762973] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101047}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762976] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762977] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847127}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762980] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762983] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200150}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762986] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762988] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847016}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762992] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762994] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200343}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.762997] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.762999] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50416}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763002] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763005] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50265}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763010] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763012] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847356}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763015] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763017] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200318}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763021] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763023] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200333}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763026] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763028] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:106}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763032] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763034] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60307}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763037] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763039] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50411}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763044] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763047] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200238}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763050] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763051] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200482}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763055] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763058] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200257}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763063] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763065] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846993}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763068] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763070] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50459}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763073] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763075] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50389}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763079] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763081] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200442}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763085] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763087] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101059}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763090] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763092] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200463}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763096] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763098] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50131}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763101] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763103] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200093}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763108] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763111] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50262}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763114] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763115] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200455}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763120] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763122] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:324}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763125] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763128] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847419}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763131] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763133] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847358}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763136] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763138] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:441}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763142] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763144] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60219}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763147] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763150] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:130}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763153] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763155] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200481}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763160] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763163] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:262}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763166] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763167] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847413}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763170] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763173] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50143}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763176] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763178] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60155}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763184] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763186] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200248}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763189] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763191] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200320}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763195] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763197] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:114}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763200] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763202] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50374}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763205] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763208] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60240}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763211] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763213] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:206}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763217] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763219] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:207}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763222] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763224] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:498}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763227] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763230] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:380}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763233] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763235] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200149}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763238] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763240] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763244] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763246] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60413}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763250] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763252] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847018}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763255] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763258] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50345}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763261] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763263] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200072}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763266] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763268] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200181}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763272] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763274] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200341}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763277] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763279] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50166}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763282] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763284] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60356}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763289] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763292] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50434}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763295] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763297] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:295}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763301] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763303] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847290}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763307] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763309] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60371}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763312] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763314] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50180}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763318] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763321] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60343}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763324] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763327] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200339}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763330] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763332] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200164}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763335] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763337] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60143}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763340] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763342] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60406}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763345] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763348] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847233}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763353] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763355] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847257}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763358] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763361] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:370}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763364] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763366] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60255}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763370] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763372] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847158}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763375] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763377] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50242}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763382] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763384] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:436}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763388] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763390] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847042}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763393] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763395] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101046}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763398] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763400] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60427}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763404] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763405] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60265}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763409] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763411] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:430}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763416] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763419] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847048}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763422] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763424] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60105}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763428] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763432] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60252}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763435] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763437] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:272}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763440] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763442] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847062}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763446] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763448] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:117}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763452] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763465] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=12] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200089}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763472] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=5] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763474] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50295}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763477] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763479] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50382}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763482] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763484] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50394}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763486] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763488] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200174}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763492] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763494] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50152}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763499] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763501] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847274}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763504] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763506] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200434}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763509] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763511] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60475}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763516] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763518] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200254}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763522] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763523] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200443}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763527] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763529] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:251}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763532] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763535] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50490}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763538] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763540] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847047}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763545] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763547] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847360}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763550] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763552] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847252}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763557] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763560] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60191}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763563] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763565] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:223}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763568] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763570] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200490}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763574] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763576] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50291}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763580] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763582] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101045}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763585] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763588] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50371}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763591] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763593] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847382}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763596] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763598] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:488}, multi_version_start=1726210103162544000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763602] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763604] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847277}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763607] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763609] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200019}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763613] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763615] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60331}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763618] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763621] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847003}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763624] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763626] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847132}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763629] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763631] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:131}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763637] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763639] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60442}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763642] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763644] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60284}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763646] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763648] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847353}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763651] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763653] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200488}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763656] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763658] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847322}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763661] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763663] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847398}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763666] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763668] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847148}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763671] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763674] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847088}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763677] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763679] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50243}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763682] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763685] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200207}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763688] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763690] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60316}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763694] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763696] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60182}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763699] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763701] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763705] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763707] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200415}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763710] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763713] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200035}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763716] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763718] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:246}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763722] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763724] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60499}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763727] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763730] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101078}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763733] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763735] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:148}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763738] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763740] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847143}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763744] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763746] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200033}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763749] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763751] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:408}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763755] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763757] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200114}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763760] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763762] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200326}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763765] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763767] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101049}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763771] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763773] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:247}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763776] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763778] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200172}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763782] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763784] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:365}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763788] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763790] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847045}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763793] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.763796] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200317}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.763799] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=2] REACH SYSLOG RATE LIMIT [frequency] [2024-09-26 16:00:20.764310] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=0] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.764313] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847079}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.764697] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=0] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.764702] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:240}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.764913] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.764916] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101025}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.765096] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=0] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.765099] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50168}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.765522] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=0] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:20.765525] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847023}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:20.765667] INFO [STORAGE] runTimerTask (ob_tenant_tablet_scheduler.cpp:174) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B95-0-0] [lt=0] SSTableGCTask(cost_ts=10454) [2024-09-26 16:00:20.770827] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:20.770837] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.779744] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:20.779750] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337620779742], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:20.779757] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337620779742], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:20.779760] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:20.779764] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:20.798181] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337620779742]}) [2024-09-26 16:00:20.801117] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27] success to notify_need_writing_throttling True [2024-09-26 16:00:20.801125] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.803090] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:20.803100] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:20.803118] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337620803088) [2024-09-26 16:00:20.803125] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337620702037, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:20.803132] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:20.803141] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:20.803144] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:20.803460] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:20.803471] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:20.803475] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:20.803479] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:20.803483] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:20.803485] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:20.803489] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:20.803492] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:20.803493] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:20.803496] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:20.803498] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:20.803500] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:20.803502] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:20.803523] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=21][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:20.803530] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:20.803532] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:20.803535] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:20.803538] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:20.803541] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:20.803544] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:20.803547] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:20.803553] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:20.803559] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:20.803561] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:20.803563] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:20.803568] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:20.803572] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:20.803575] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885728-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:20.803579] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:20.803582] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:20.803584] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:20.803587] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337620803380, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:20.803592] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:20.803595] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:20.803624] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:20.803629] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337620803629, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=492, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:20.803636] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:20.803640] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:20.803657] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223350039538, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.803661] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:20.803668] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:20.803673] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:20.803677] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:20.803683] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:20.803684] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:20.803687] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337620803644) [2024-09-26 16:00:20.820950] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:20.820963] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:20.820954] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:20.820971] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:20.821616] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:20.821627] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:20.821619] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:20.821634] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:20.821640] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:20.821646] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:20.821655] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:20.831456] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:20.831462] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.855831] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=12][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337621785135, now=[mts=1727337619855350]) [2024-09-26 16:00:20.855842] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337621785135, now=[mts=1727337619855350], now0=[mts=1727337619855350], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:20.855848] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a311c660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500640, assoc_session_id:3221500640, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337619854829, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:20.855866] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a2319bf8, id:3221500640, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a311c660}) [2024-09-26 16:00:20.855873] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500640, *tx_desc={this:0x7f92a311c660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500640, assoc_session_id:3221500640, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337619854829, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:20.855881] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500640, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a311c660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500640, assoc_session_id:3221500640, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337619854829, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337619855328, use_das=false, nested_level=0, session={this:0x7f92a2319bf8, id:3221500640, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a311c660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:20.855897] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:20.855902] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:20.855912] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=9][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:20.855914] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:20.855917] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:20.855921] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:20.855926] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=999209, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337621855135) [2024-09-26 16:00:20.855930] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:20.855937] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:20.855939] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:20.855941] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:20.855946] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:20.855981] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:20.855986] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:20.861672] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29] success to notify_need_writing_throttling True [2024-09-26 16:00:20.861682] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.888042] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=4] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-09-26 16:00:20.888064] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=18] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=684) [2024-09-26 16:00:20.889016] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:658) [2687][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=17] dump dest keepalive data states, addr: "192.168.0.45:2882", last_write_ts_=1727337620688739, last_read_ts_=1727337620480785, in_black_=0 [2024-09-26 16:00:20.891965] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:20.891975] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.891984] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:20.892063] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=16646144, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:20.892095] INFO [COMMON] wash (ob_kvcache_store.cpp:335) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash memory, (tenant_id=1, cache_size=16646144, lower_mem_limit=2147483648, upper_mem_limit=2147483648, min_wash_size=0, max_wash_size=0, mem_usage=1305686016, reserve_mem=137180449, wash_size=2254405632) [2024-09-26 16:00:20.892189] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash time detail, (compute_wash_size_time=56, refresh_score_time=14, wash_time=11) [2024-09-26 16:00:20.892800] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1289737, clean_num=31457) [2024-09-26 16:00:20.898773] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337620779742]}) [2024-09-26 16:00:20.903733] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:20.903741] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337620903730) [2024-09-26 16:00:20.903746] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337620803129, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:20.903780] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223350140202, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:20.903789] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:20.903796] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:20.903801] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:20.903803] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:20.903810] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:20.903812] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:20.903814] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337620903749) [2024-09-26 16:00:20.919516] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:20.922267] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:20.922274] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.944694] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: SHARED_BLOCK_ReaderWriter_IO, mode: read, size: 4096.00, iops: 56.72, rt: 2378.77) [2024-09-26 16:00:20.944703] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:20.944720] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 12070.00) [2024-09-26 16:00:20.944723] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:20.944728] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:20.944731] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:20.952500] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:20.952505] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.962277] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:20.978919] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337620779742]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:20.978929] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=10][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:20.982788] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:20.982796] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:20.983967] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] new mysql sessid created(conn.sessid_=3221500645, support_ssl=false) [2024-09-26 16:00:20.984002] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] sm conn init succ(conn.sessid_=3221500645, sess.client_addr_="127.0.0.1:36302") [2024-09-26 16:00:20.984010] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500645, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:20.984039] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:20.984049] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:20.984085] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F97-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:20.984293] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F97-0-0] [lt=10][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:20.984302] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F97-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"_?\1!MA=OWF7"Chc;8IE"}, ret=-4043) [2024-09-26 16:00:20.984308] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F97-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:20.984315] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F97-0-0] [lt=6][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:20.984325] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F97-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:20.984347] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F97-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500645, proxy_sessid:0}) [2024-09-26 16:00:20.984352] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F97-0-0] [lt=4] mark session id unused(sessid=3221500645) [2024-09-26 16:00:20.984356] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F97-0-0] [lt=3][errcode=0] server close connection(sessid=3221500645, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:20.984359] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F97-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500645, proxy_sessid=0) [2024-09-26 16:00:20.984362] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F97-0-0] [lt=3][errcode=-4016] session is null [2024-09-26 16:00:20.984364] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F97-0-0] [lt=1] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500645, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:20.984396] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500645, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337620983836, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:20.984406] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] kill and revert session(conn.sessid_=3221500645, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:20.984409] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500645, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337620983836, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:20.984414] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500645, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:20.999392] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337620779742]}) [2024-09-26 16:00:20.999440] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D7B8-0-0] [lt=8][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337620999385}) [2024-09-26 16:00:21.004310] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:21.004321] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337621004307) [2024-09-26 16:00:21.004337] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337620903749, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:21.004353] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223350240716, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:21.004360] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:21.004367] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:21.004372] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:21.004375] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:21.004380] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.004383] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.004385] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337621004341) [2024-09-26 16:00:21.013089] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16] success to notify_need_writing_throttling True [2024-09-26 16:00:21.013097] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.023161] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:21.023172] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:21.023176] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:21.023180] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:21.023183] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:21.023186] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:21.023189] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:21.023193] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:21.023195] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:21.023197] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:21.023199] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:21.023201] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:21.023203] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:21.023205] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:21.023212] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:21.023215] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:21.023217] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:21.023219] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:21.023222] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:21.023224] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:21.023227] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:21.023232] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:21.023237] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:21.023241] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:21.023242] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:21.023248] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:21.023251] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:21.023254] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:21.023278] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586E-0-0] [lt=23][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:21.023281] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586E-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:21.023283] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:21.023286] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586E-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337621023088, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:21.023289] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586E-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:21.023291] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586E-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:21.023314] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586E-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:21.023318] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586E-0-0] [lt=4][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:21.023425] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E2-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:21.023433] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E2-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:21.023444] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E2-0-0] [lt=10][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:21.023466] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E2-0-0] [lt=22][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:21.033587] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=6] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:21.033872] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=6] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:21.033884] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:21.033887] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:21.033890] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:21.033893] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:21.033895] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:21.033898] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:21.033901] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:21.033905] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:21.033908] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:21.033910] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:21.033912] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:21.033914] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:21.033916] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:21.033922] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:21.033925] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:21.033930] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:21.033932] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:21.033934] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:21.033937] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:21.033939] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:21.033944] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:21.033948] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:21.033950] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:21.033952] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:21.033957] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:21.033960] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:21.033963] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2582][RSAsyncTask3][T0][YB42C0A8002D-0006230057684E52-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:21.033966] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:21.033968] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:21.033972] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:21.033975] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337621033824, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:21.033978] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:21.034040] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:21.039457] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=11][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337620779742]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:21.039466] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=8][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:21.044337] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:21.044365] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.074583] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:21.074590] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.078486] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=26] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:21.093290] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:21.093391] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=58, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:21.094014] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=1321194, clean_num=31457) [2024-09-26 16:00:21.100981] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=43] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337620779742]}) [2024-09-26 16:00:21.104879] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:21.104886] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:21.104891] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337621104876) [2024-09-26 16:00:21.104891] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.104896] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337621004341, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:21.104912] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223350341374, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:21.104927] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:21.104937] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:21.104942] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:21.104945] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:21.104951] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.104954] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.104955] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337621104899) [2024-09-26 16:00:21.119539] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:21.135156] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:21.135164] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.165460] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:21.165468] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.197844] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:21.197852] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.197889] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:21.197903] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:21.197908] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:21.201560] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337620779742]}) [2024-09-26 16:00:21.205503] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:21.205532] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337621205497) [2024-09-26 16:00:21.205539] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337621104899, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:21.205564] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223350442008, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:21.205572] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:21.205582] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:21.205587] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:21.205592] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:21.205600] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.205603] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.205607] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337621205543) [2024-09-26 16:00:21.219190] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1727337621219188, bytes: 14562202, bw: 0.003614 MB/s, add_ts: 1008677, add_bytes: 3822 [2024-09-26 16:00:21.223405] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:21.223456] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=49][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:21.223462] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:21.223466] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:21.223470] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:21.223472] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:21.223475] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:21.223478] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:21.223480] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:21.223482] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:21.223485] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:21.223487] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:21.223489] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:21.223492] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:21.223509] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:21.223512] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:21.223516] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:21.223518] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:21.223520] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:21.223523] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:21.223526] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:21.223532] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:21.223538] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:21.223540] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:21.223541] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:21.223548] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:21.223552] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:21.223555] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:21.223561] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:21.223564] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:21.223566] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:21.223568] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337621223334, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:21.223572] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:21.223575] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:21.223614] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=3][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:21.223618] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=4][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:21.223621] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=3][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:21.223623] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:21.223626] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:21.223629] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:21.223631] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587D-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:21.228001] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:21.228010] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.258256] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:21.258265] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.266077] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D338F-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:21.280795] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.280807] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337621280791], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:21.280827] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337621280791], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:21.280833] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=5102, get_gts_with_stc_cnt=11888, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-09-26 16:00:21.284592] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337622213793, now=[mts=1727337620284061]) [2024-09-26 16:00:21.284600] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337622213793, now=[mts=1727337620284061], now0=[mts=1727337620284061], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:21.284606] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92690042e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337620283459, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:21.284623] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92690042e0}) [2024-09-26 16:00:21.284633] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=9][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92690042e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337620283459, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:21.284640] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92690042e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337620283459, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337620284033, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92690042e0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:21.284667] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=26][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:21.284672] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:21.284674] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:21.284677] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:21.284679] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:21.284683] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:21.284688] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=999105, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337622283793) [2024-09-26 16:00:21.284691] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:21.284698] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:21.284700] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:21.284701] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:21.284706] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:21.284745] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:00:21.284747] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:21.288431] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:21.288437] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.295487] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:21.295588] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=49, refresh_score_time=13, wash_time=2) [2024-09-26 16:00:21.296174] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=1352651, clean_num=31457) [2024-09-26 16:00:21.302250] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337621280791]}) [2024-09-26 16:00:21.306111] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:21.306124] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337621306109) [2024-09-26 16:00:21.306129] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337621205542, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:21.306134] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:21.306153] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:21.306157] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:21.306487] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:21.306512] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=23][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:21.306516] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:21.306521] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:21.306525] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:21.306527] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:21.306531] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:21.306534] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:21.306537] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:21.306541] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=4][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:21.306544] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:21.306546] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:21.306548] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:21.306550] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:21.306558] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:21.306560] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:21.306563] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:21.306566] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:21.306571] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:21.306574] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:21.306576] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:21.306582] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:21.306587] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:21.306590] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:21.306591] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:21.306596] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:21.306600] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:21.306603] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885729-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:21.306605] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:21.306609] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:21.306611] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:21.306614] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337621306404, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:21.306618] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:21.306620] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:21.306649] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:21.306652] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337621306651, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=513, wlock_time=21, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:21.306659] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:21.306671] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:21.306688] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223350542565, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:21.306693] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:21.306699] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:21.306704] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:21.306706] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:21.306713] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.306715] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.306717] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337621306674) [2024-09-26 16:00:21.318469] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:21.318478] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.319783] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:21.348552] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:21.348560] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.378653] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:21.378665] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.381255] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=9] current monitor number(seq_=-1) [2024-09-26 16:00:21.382197] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:21.384317] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [2645][Blacklist][T0][Y0-0000000000000000-0-0] [lt=8] blacklist_loop exec finished(cost_time=9, is_enabled=true, send_cnt=0) [2024-09-26 16:00:21.402720] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337621280791]}) [2024-09-26 16:00:21.406359] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:21.406397] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337621406355) [2024-09-26 16:00:21.406404] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337621306131, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:21.406425] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223350642790, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:21.406478] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=52] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:21.406487] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:21.406493] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:21.406495] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:21.406502] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.406508] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.406510] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337621406408) [2024-09-26 16:00:21.408712] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:21.408721] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.437909] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=23] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.437925] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=15] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.437930] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.438558] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.438566] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.438570] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.438880] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:21.438891] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.439936] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.439946] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.439952] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.439955] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.439961] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.440496] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.440502] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.440509] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.440516] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.440519] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.440521] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.440526] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.440528] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.440530] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.441829] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.441840] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.441843] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.441847] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.441848] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.441850] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.441852] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.441854] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.441856] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.442833] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.442838] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.442840] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.442856] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.442859] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.442862] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.442864] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.442867] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.442869] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.444850] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.444855] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.444859] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.444861] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.444863] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.444865] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.445495] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.445500] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.445504] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.445505] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:21.445508] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.446701] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.446710] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.446714] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.447691] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.447696] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.447699] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.448674] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.448678] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.448681] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.449601] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.449608] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.449611] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.450187] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.450192] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.450194] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.451591] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.451604] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.451608] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.452118] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.452126] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.452145] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.453565] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.453582] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.453587] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.453590] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.454521] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.454528] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.454531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.455150] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.455157] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.455161] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.456467] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.456473] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.456477] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.457117] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.457126] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.457130] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.458423] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.458459] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.458463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.459059] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.459069] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.459073] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=34][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.459587] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.459593] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.459597] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.461356] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.461388] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.461414] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.462334] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.462340] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.462343] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=29][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.463311] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.463318] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.463322] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.464309] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.464322] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.464337] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.464341] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.465269] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.465275] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.465279] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.466246] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.466251] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.466254] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.467216] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.467223] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.467226] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.469183] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.469193] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.469197] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.469184] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:21.469207] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.471170] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.471180] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.471185] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.472099] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.472106] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.472109] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.472718] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.472722] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.472725] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.474061] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.474068] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.474071] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:21.493616] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] ====== tenant freeze timer task ====== [2024-09-26 16:00:21.493630] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:21.493663] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337620617622, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337620617589, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337620617639, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:21.493686] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:21.493731] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851CF-0-0] [lt=4] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:21.493741] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851CF-0-0] [lt=7] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:21.493746] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851CF-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:21.493749] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851CF-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:21.493753] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851CF-0-0] [lt=4][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:21.493756] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851CF-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:21.493758] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851CF-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:21.493767] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851CF-0-0] [lt=9] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:21.493769] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851CF-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:21.493771] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851CF-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:21.493774] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851CF-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:21.493797] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851CF-0-0] [lt=4] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:21.496432] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:21.496567] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=48, refresh_score_time=14, wash_time=28) [2024-09-26 16:00:21.497169] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1384108, clean_num=31457) [2024-09-26 16:00:21.499472] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:21.499481] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.503130] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337621280791]}) [2024-09-26 16:00:21.506415] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223350742839, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:21.506427] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:21.506445] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:21.506451] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:21.506487] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:21.506502] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.506506] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.506509] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337621506400) [2024-09-26 16:00:21.519959] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:21.526973] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E3-0-0] [lt=37][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:21.526983] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E3-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:21.526992] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E3-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:21.526997] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E3-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:21.529720] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:21.529728] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.536566] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:171) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] Eliminate task evict sql audit(request_manager_->get_tenant_id()=1, queue_size=100000, config_mem_limit_=64424509, request_manager_->get_size_used()=6241, evict_high_size_level=90000, evict_low_size_level=80000, allocator->allocated()=10398720, evict_high_mem_level=32212254, evict_low_mem_level=0) [2024-09-26 16:00:21.536577] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6241, mem_used=10398720) [2024-09-26 16:00:21.558103] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=12] gc stale ls task succ [2024-09-26 16:00:21.558139] WDIAG [MVCC] study (ob_multi_version_garbage_collector.cpp:319) [2953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4012] study min unallocated GTS failed(ret=-4012) [2024-09-26 16:00:21.558144] INFO [MVCC] study (ob_multi_version_garbage_collector.cpp:391) [2953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] study multi version garabage collector end(ret=-4012, this={this:0x7f92b8cf8e80, last_study_timestamp:0, last_refresh_timestamp:0, last_reclaim_timestamp:1727335461821252, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:true, gc_is_disabled:false, global_reserved_snapshot:{val:0, v:0}, is_inited:true}, min_unallocated_GTS={val:4611686018427387903, v:0}, min_unallocated_GTS={val:4611686018427387903, v:0}, max_committed_txn_version={val:4611686018427387903, v:0}, min_active_txn_version={val:4611686018427387903, v:0}) [2024-09-26 16:00:21.558153] WDIAG [MVCC] repeat_study (ob_multi_version_garbage_collector.cpp:186) [2953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4012] repeat study failed, we will retry immediately(ret=-4012, this={this:0x7f92b8cf8e80, last_study_timestamp:0, last_refresh_timestamp:0, last_reclaim_timestamp:1727335461821252, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:true, gc_is_disabled:false, global_reserved_snapshot:{val:0, v:0}, is_inited:true}, current_timestamp=1727337620558004) [2024-09-26 16:00:21.558513] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_reserved_snapshot, table_name.ptr()="data_size:23, data:5F5F616C6C5F72657365727665645F736E617073686F74", ret=-5019) [2024-09-26 16:00:21.558524] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_reserved_snapshot, ret=-5019) [2024-09-26 16:00:21.558528] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_reserved_snapshot, db_name=oceanbase) [2024-09-26 16:00:21.558531] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_reserved_snapshot) [2024-09-26 16:00:21.558535] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:21.558536] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:21.558540] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_reserved_snapshot' doesn't exist [2024-09-26 16:00:21.558543] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:21.558546] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:21.558548] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:21.558551] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:21.558555] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:21.558558] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:21.558560] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:21.558566] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:21.558569] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:21.558574] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:21.558577] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:21.558579] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 , ret=-5019) [2024-09-26 16:00:21.558582] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:21.558584] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:" select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1 "}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:21.558591] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:21.558596] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=4][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:21.558604] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B75-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:21.558861] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B76-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_reserved_snapshot, table_name.ptr()="data_size:23, data:5F5F616C6C5F72657365727665645F736E617073686F74", ret=-5019) [2024-09-26 16:00:21.558879] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2953][T1_Occam][T1][YB42C0A8002D-0006230058B84B76-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:21.558894] INFO [MVCC] reclaim (ob_multi_version_garbage_collector.cpp:1009) [2953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=0] reclaim multi version garabage collector end(this={this:0x7f92b8cf8e80, last_study_timestamp:0, last_refresh_timestamp:0, last_reclaim_timestamp:1727335461821252, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:true, gc_is_disabled:false, global_reserved_snapshot:{val:0, v:0}, is_inited:true}, ret=-5019, role=1, reclaimable_servers=[], snapshot_servers=[], is_this_server_disabled=false) [2024-09-26 16:00:21.558903] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [2953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] run this task use too much time(*p_task_={this:0x7f92be3ba3b0, caller:ob_multi_version_garbage_collector.cpp:start:115, function_type:ZN9oceanbase19concurrency_control30ObMultiVersionGarbageCollector5startEvE5$_765, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f92be3ba610, time_wheel:0x7f92af4a2850, thread_pool:0x7f92af4a20b0, is_running:0, time_interval:60.00s, expected_run_time:16:01:19.311, task_priority:1, with_handle_protected:1}) [2024-09-26 16:00:21.558916] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:403) [2953][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (*this=|threshold=1.00s|start at 16:00:21.558|total=1.00s) [2024-09-26 16:00:21.560089] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:21.560097] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.586392] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:21.590280] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:21.590286] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.602005] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.602009] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.602020] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.602133] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.603974] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337621280791]}) [2024-09-26 16:00:21.606910] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:21.606922] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:21.606942] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337621606920) [2024-09-26 16:00:21.606947] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337621406408, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:21.606963] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223350842389, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:21.606969] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:21.606977] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:21.606982] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:21.606985] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:21.606991] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.606993] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.606995] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337621606951) [2024-09-26 16:00:21.620558] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:21.620563] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.630821] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855D5-0-0] [lt=3] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:21.641973] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4184, dropped:506, tid:2883}]) [2024-09-26 16:00:21.651835] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:21.651841] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.651874] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=4] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:21.665493] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=5] PNIO [ratelimit] time: 1727337621665492, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008353, add_bytes: 0 [2024-09-26 16:00:21.682145] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:21.682151] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.697825] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:21.697938] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash time detail, (compute_wash_size_time=61, refresh_score_time=16, wash_time=2) [2024-09-26 16:00:21.698538] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1415565, clean_num=31457) [2024-09-26 16:00:21.704398] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337621280791]}) [2024-09-26 16:00:21.706966] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:21.706975] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337621706963) [2024-09-26 16:00:21.706980] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337621606951, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:21.706998] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223350942977, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:21.707004] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:21.707011] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:21.707016] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:21.707019] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:21.707026] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.707028] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.707030] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337621706984) [2024-09-26 16:00:21.712348] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:21.712357] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.713328] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337620711900]) [2024-09-26 16:00:21.713339] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337620711900], now0=[mts=1727337620711900], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:21.713347] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337620710554, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:21.713364] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:21.713372] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337620710554, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:21.713379] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337620710554, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:21.713396] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:21.713401] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:21.713404] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:21.713407] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:21.713410] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:21.713414] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=17, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:21.713419] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] will sleep(sleep_us=17000, remain_us=11838409, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:21.720786] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:21.730926] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:17, local_retry_times:17, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:21.730953] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=22][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:21.730957] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:21.730958] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:21.730964] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:21.731004] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:21.731007] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:21.731913] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=6] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:21.742683] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:21.742695] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.772942] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=26] success to notify_need_writing_throttling True [2024-09-26 16:00:21.772948] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.781832] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:21.781843] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337621781829], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:21.781851] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337621781829], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:21.781855] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:21.781858] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:21.798403] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [2915][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=8] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-09-26 16:00:21.803249] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:21.803254] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.805181] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337621781829]}) [2024-09-26 16:00:21.807109] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:21.807117] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:21.807141] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337621807107) [2024-09-26 16:00:21.807146] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337621706984, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:21.807152] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:21.807160] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:21.807163] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:21.807517] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572A-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:21.807526] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572A-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:21.807554] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572A-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:21.807587] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337621807586, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=430, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:21.807596] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:21.807613] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223351043248, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:21.807617] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:21.807623] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:21.807628] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:21.807630] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:21.807634] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.807636] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.807638] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337621807600) [2024-09-26 16:00:21.820797] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.820807] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.820814] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.820822] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.821403] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.821415] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.821423] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.821444] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.821447] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.821454] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.821460] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:21.833304] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:21.833310] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.842807] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:81, tid:2932}]) [2024-09-26 16:00:21.856944] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=5][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337621785135, now=[mts=1727337620855980]) [2024-09-26 16:00:21.856953] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337621785135, now=[mts=1727337620855980], now0=[mts=1727337620855980], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:21.856959] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=4][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a311c660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500640, assoc_session_id:3221500640, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337620855905, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:21.856975] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a2319bf8, id:3221500640, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a311c660}) [2024-09-26 16:00:21.856982] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500640, *tx_desc={this:0x7f92a311c660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500640, assoc_session_id:3221500640, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337620855905, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:21.856990] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500640, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a311c660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500640, assoc_session_id:3221500640, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337620855905, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337619855328, use_das=false, nested_level=0, session={this:0x7f92a2319bf8, id:3221500640, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a311c660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:21.857006] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:21.857010] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:21.857013] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:21.857015] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:21.857017] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:21.857020] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:21.857024] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:21.857029] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:21.857031] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=1][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:21.857032] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:21.857036] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:21.857038] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2001710) [2024-09-26 16:00:21.857041] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507B-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:21.857043] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:21.857046] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:21.857048] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:21.857050] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8900, start=1727337619855324, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:21.857053] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:00:21.857056] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:21.857083] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:21.857085] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:21.857087] WDIAG [SHARE] get_snapshot_gc_scn (ob_global_stat_proxy.cpp:164) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:21.857103] WDIAG [STORAGE] get_global_info (ob_tenant_freeze_info_mgr.cpp:811) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4012] fail to get global info(ret=-4012, tenant_id=1) [2024-09-26 16:00:21.857106] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:954) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] failed to get global info(ret=-4012) [2024-09-26 16:00:21.857108] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1008) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to try update info(tmp_ret=-4012, tmp_ret="OB_TIMEOUT") [2024-09-26 16:00:21.857113] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1044) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2] success to update min reserved snapshot(reserved_snapshot=0, duration=1800, snapshot_gc_ts_=0) [2024-09-26 16:00:21.857116] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1071) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2] update reserved snapshot finished(cost_ts=3, reserved_snapshot=0) [2024-09-26 16:00:21.863771] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:21.863788] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.894076] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:21.894083] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.894091] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:21.899009] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:21.899119] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=50, refresh_score_time=17, wash_time=1) [2024-09-26 16:00:21.899725] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=1447022, clean_num=31457) [2024-09-26 16:00:21.905768] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337621781829]}) [2024-09-26 16:00:21.907736] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:21.907744] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337621907733) [2024-09-26 16:00:21.907748] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337621807149, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:21.907763] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223351143296, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:21.907768] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:21.907775] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:21.907779] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:21.907781] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:21.907787] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.907789] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:21.907791] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337621907751) [2024-09-26 16:00:21.921393] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:21.924266] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:21.924273] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.951725] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 12089.00) [2024-09-26 16:00:21.951736] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:21.951748] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:21.951751] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:21.954532] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:21.954548] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.962419] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] tx gc loop thread is running(MTL_ID()=1) [2024-09-26 16:00:21.962424] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] try gc retain ctx [2024-09-26 16:00:21.962857] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-09-26 16:00:21.962881] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:21.984805] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:21.984821] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:21.985002] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500651, support_ssl=false) [2024-09-26 16:00:21.985025] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221500651, sess.client_addr_="127.0.0.1:36304") [2024-09-26 16:00:21.985032] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f9262e04030, session_id:3221500651, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:21.985051] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:21.985060] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=5] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:21.985112] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F99-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:21.985284] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F99-0-0] [lt=5][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:21.985290] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F99-0-0] [lt=5][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"|]:KImiGfN1Ja*K|OS6S"}, ret=-4043) [2024-09-26 16:00:21.985296] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F99-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:21.985299] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F99-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:21.985306] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F99-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:21.985327] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F99-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500651, proxy_sessid:0}) [2024-09-26 16:00:21.985331] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F99-0-0] [lt=3] mark session id unused(sessid=3221500651) [2024-09-26 16:00:21.985334] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F99-0-0] [lt=2][errcode=0] server close connection(sessid=3221500651, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:21.985337] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F99-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500651, proxy_sessid=0) [2024-09-26 16:00:21.985339] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F99-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:21.985342] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F99-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500651, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:21.985381] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4015] close sql sock by user req(*s={this:0x7f9262e04030, session_id:3221500651, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337621983999, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:21.985390] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500651, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:21.985393] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9262e04030, session_id:3221500651, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337621983999, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:21.985397] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500651, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:22.006382] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337621781829]}) [2024-09-26 16:00:22.006403] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D7C4-0-0] [lt=7][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337622006362}) [2024-09-26 16:00:22.007911] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:22.007920] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337622007908) [2024-09-26 16:00:22.007925] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337621907751, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:22.007942] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223351242887, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:22.007947] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:22.007954] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:22.007959] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:22.007962] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:22.007968] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.007970] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.007972] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337622007928) [2024-09-26 16:00:22.015168] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:22.015175] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.024249] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:22.024262] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:22.024267] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:22.024270] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:22.024274] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:22.024276] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:22.024279] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:22.024281] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:22.024283] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:22.024285] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:22.024287] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:22.024289] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:22.024292] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:22.024294] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:22.024316] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=18][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:22.024319] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:22.024322] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:22.024324] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:22.024326] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:22.024328] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:22.024334] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:22.024351] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=15][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:22.024358] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:22.024360] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:22.024362] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:22.024368] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:22.024371] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8586F-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:22.024375] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586F-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:22.024394] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586F-0-0] [lt=18][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:22.024397] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586F-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:22.024400] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586F-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:22.024402] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586F-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337622024179, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:22.024406] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586F-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:22.024411] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586F-0-0] [lt=5][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:22.024438] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586F-0-0] [lt=4][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:22.024441] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8586F-0-0] [lt=3][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:22.027061] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E4-0-0] [lt=38][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:22.027068] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E4-0-0] [lt=6][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:22.027074] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E4-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:22.027078] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E4-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:22.034694] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=8] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:22.034996] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=6] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:22.035007] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:22.035011] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:22.035014] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:22.035018] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:22.035020] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:22.035023] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:22.035025] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:22.035027] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:22.035029] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:22.035031] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:22.035033] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:22.035035] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:22.035037] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:22.035042] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:22.035045] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:22.035050] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:22.035053] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:22.035055] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:22.035057] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:22.035059] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:22.035064] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:22.035069] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:22.035071] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:22.035073] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:22.035078] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:22.035081] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F36-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:22.035084] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F36-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:22.035087] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:22.035090] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:22.035092] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:22.035095] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337622034939, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:22.035098] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:22.035147] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:22.045464] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:22.045471] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.075736] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:22.075745] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.078624] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:22.100298] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:22.100414] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:22.101033] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1478479, clean_num=31457) [2024-09-26 16:00:22.106003] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:22.106012] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.106964] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337621781829]}) [2024-09-26 16:00:22.107940] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:22.107948] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337622107938) [2024-09-26 16:00:22.107953] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337622007928, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:22.107969] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223351343606, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:22.107974] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:22.107981] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:22.107985] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:22.107988] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:22.107994] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.107996] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.107998] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337622107956) [2024-09-26 16:00:22.121613] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:22.136260] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:22.136267] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.166530] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:22.166537] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.184126] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=12][errcode=-4018] try free send task failed(ret=-4018) [2024-09-26 16:00:22.194398] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] ====== check clog disk timer task ====== [2024-09-26 16:00:22.194407] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=7] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:22.194415] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:22.196824] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:22.196829] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.207150] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337621781829]}) [2024-09-26 16:00:22.208551] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:22.208561] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337622208549) [2024-09-26 16:00:22.208565] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337622107956, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:22.208580] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223351444291, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:22.208586] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:22.208594] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:22.208598] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:22.208601] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:22.208607] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.208610] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.208612] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337622208569) [2024-09-26 16:00:22.224311] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:22.224324] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:22.224328] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:22.224346] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:22.224350] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:22.224352] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:22.224355] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:22.224357] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:22.224361] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:22.224362] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:22.224367] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=5][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:22.224370] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:22.224372] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:22.224374] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:22.224382] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:22.224385] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:22.224390] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:22.224393] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:22.224395] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:22.224398] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:22.224401] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:22.224407] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:22.224412] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:22.224415] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:22.224417] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:22.224423] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:22.224426] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:22.224429] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:22.224434] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:22.224437] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:22.224439] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:22.224441] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337622224244, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:22.224446] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:22.224448] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:22.224492] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=3][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:22.224495] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=3][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:22.224498] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=3][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:22.224500] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:22.224503] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:22.224507] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=3][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:22.224509] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587E-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:22.226104] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=15] PNIO [ratelimit] time: 1727337622226103, bytes: 14565742, bw: 0.003353 MB/s, add_ts: 1006915, add_bytes: 3540 [2024-09-26 16:00:22.227145] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:22.227158] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.257372] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:22.257390] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.267747] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D33EF-0-0] [lt=12][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:22.282893] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.282909] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337622282891], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:22.282916] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337622282891], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:22.284209] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:262) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=7] ObArchiveSender is running(thread_index=0) [2024-09-26 16:00:22.286671] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337622213793, now=[mts=1727337621284743]) [2024-09-26 16:00:22.286678] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337622213793, now=[mts=1727337621284743], now0=[mts=1727337621284743], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:22.286695] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=15][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92690042e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337621284581, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:22.286712] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92690042e0}) [2024-09-26 16:00:22.286732] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=19][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92690042e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337621284581, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:22.286739] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92690042e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337621284581, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337620284033, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92690042e0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:22.286757] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:22.286762] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:22.286765] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:22.286769] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:22.286772] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:22.286775] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:22.286779] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:22.286786] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:22.286788] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:22.286789] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:22.286794] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:22.286798] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2002764) [2024-09-26 16:00:22.286801] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:22.286804] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:22.286806] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:22.286809] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:22.286811] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:22.286817] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=4][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:22.286843] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:22.286847] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:22.286849] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:22.286852] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:22.286856] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:22.286859] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:22.286979] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDC-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:22.286989] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:22.286992] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:22.287007] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:22.287024] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=16][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:22.287290] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:00:22.287297] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:22.287560] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:22.287567] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.301492] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:22.301592] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=61, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:22.302188] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=7, clean_time=0, clean_start_pos=1509936, clean_num=31457) [2024-09-26 16:00:22.307225] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337622282891]}) [2024-09-26 16:00:22.309168] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:22.309189] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337622309165) [2024-09-26 16:00:22.309194] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337622208568, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:22.309210] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:22.309219] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:22.309223] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:22.309533] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:22.309540] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:22.309543] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:22.309546] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:22.309549] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:22.309551] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:22.309555] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:22.309557] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:22.309562] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:22.309564] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:22.309566] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:22.309568] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:22.309571] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:22.309573] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:22.309579] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:22.309582] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:22.309585] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:22.309589] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:22.309592] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:22.309595] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:22.309597] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:22.309601] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:22.309605] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:22.309607] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:22.309608] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:22.309613] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:22.309617] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:22.309619] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572B-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:22.309622] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:22.309625] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:22.309627] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:22.309630] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337622309470, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:22.309633] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:22.309635] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:22.309658] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:22.309662] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337622309661, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=446, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:22.309668] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:22.309681] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:22.309699] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1}, delta=223351545619, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:22.309704] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:22.309710] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:22.309714] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:22.309719] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:22.309724] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.309726] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.309727] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337622309688) [2024-09-26 16:00:22.317974] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:22.317982] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.322133] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=597664, replace_num=15728) [2024-09-26 16:00:22.322144] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:22.349218] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:22.349236] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.369003] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:22.369016] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:22.369022] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=5][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:22.369026] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:22.369029] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:22.369031] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:22.369033] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:22.369035] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:22.369038] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:22.369040] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:22.369043] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:22.369045] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:22.369047] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:22.369051] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:22.369054] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:22.369056] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:22.369062] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:22.369065] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:22.369068] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:22.369071] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:22.369073] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-09-26 16:00:22.369079] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=2][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:22.369084] WDIAG [SERVER] do_process (obmp_query.cpp:771) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=4][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-09-26 16:00:22.369090] WDIAG [SERVER] do_process (obmp_query.cpp:903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, session={this:0x7f92a230bdf8, id:3221487638, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-09-26 16:00:22.369100] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F48-0-0] [lt=6] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:22.379530] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:22.379537] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.382413] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=11] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:22.407829] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337622282891]}) [2024-09-26 16:00:22.409758] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:22.409766] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:22.409765] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.409776] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337622409764) [2024-09-26 16:00:22.409782] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337622309208, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:22.409797] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223351646209, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:22.409803] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:22.409819] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:22.409826] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:22.409829] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:22.409835] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.409838] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.409839] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337622409784) [2024-09-26 16:00:22.438084] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.439065] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.439084] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.440004] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:22.440015] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.440046] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.440051] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.441003] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=23] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.441009] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.441011] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.441018] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.441017] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.441022] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.441573] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.441579] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.441581] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.441630] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.441634] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.441638] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.442106] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.442111] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.442113] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.442172] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.442176] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.442180] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.442951] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.442967] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.442973] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.443030] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.443033] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.443036] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.443503] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.443507] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.443509] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.443709] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.443715] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.443718] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.444216] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.444221] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.444223] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.444304] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.444308] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.444897] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.444902] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.445049] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.445056] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.445598] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.445603] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.445671] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.445677] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:22.446242] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.446285] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.446806] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.446808] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.447753] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.447767] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.448283] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.448340] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.448817] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.448997] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.449492] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.449835] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.450542] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.450541] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.451639] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.451694] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.452725] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.452865] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.453308] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.453822] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.453844] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.454672] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.454675] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.455647] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.455685] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.456638] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.456642] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.458601] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.458643] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.459598] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.459604] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.460516] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.460561] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.461574] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.461628] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.462498] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.462533] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.464465] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.464502] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.465463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.465471] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.466375] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.466390] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.467378] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.467405] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.468337] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.468348] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.469347] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.469350] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.470320] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:22.470319] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.470337] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.470360] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.472269] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.472303] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.473233] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.474196] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.474218] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.475234] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.475239] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.476156] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.476168] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.478099] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.478120] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.479097] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.479098] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.480102] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.480102] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.481044] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.481047] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.482016] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.482039] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.483985] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.483988] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.484957] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.484949] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.486897] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.486943] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.488862] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.488881] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=36][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.490941] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.490958] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.491758] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.491759] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.492763] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.492767] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:22.501608] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:22.501615] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.502657] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:22.502769] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=24, wash_time=2) [2024-09-26 16:00:22.503373] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=1541393, clean_num=31457) [2024-09-26 16:00:22.508467] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337622282891]}) [2024-09-26 16:00:22.508465] INFO [ARCHIVE] produce_log_fetch_task_ (ob_archive_sequencer.cpp:174) [2939][T1_ArcSeq][T1][YB42C0A8002D-0006230055D84B24-0-0] [lt=10] archive round not in doing status, just skip(key={incarnation:-1, dest_id:-1, round:-1}, state={status:"INVALID"}) [2024-09-26 16:00:22.510366] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:22.510375] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337622510363) [2024-09-26 16:00:22.510380] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337622409784, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:22.510398] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223351745964, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:22.510405] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:22.510412] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:22.510417] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:22.510420] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:22.510426] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.510428] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.510430] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337622510383) [2024-09-26 16:00:22.522187] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=4] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:22.524162] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E5-0-0] [lt=24][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:22.524172] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E5-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:22.524181] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E5-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:22.524186] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E5-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:22.532837] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:22.532843] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.536718] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=4] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=1, size_used=6252, mem_used=10398720) [2024-09-26 16:00:22.550442] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=17] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:22.558234] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=8] gc stale ls task succ [2024-09-26 16:00:22.563081] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:22.563087] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.566457] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=7] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:22.566469] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:22.566473] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:22.566499] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=26][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:22.566504] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:22.566511] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:22.566515] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:22.566517] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:22.566519] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:22.566521] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:22.566523] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:22.566526] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:22.566528] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:22.566530] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:22.566537] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:22.566540] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:22.566543] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:22.566548] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:22.566550] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:22.566553] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:22.566555] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:22.566569] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DCF-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:22.591100] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:22.593162] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:22.593170] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.601193] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.602178] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.602257] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.602270] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.609046] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337622282891]}) [2024-09-26 16:00:22.610997] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:22.611023] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:22.611032] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337622611021) [2024-09-26 16:00:22.611037] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337622510383, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:22.611055] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223351847464, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:22.611060] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:22.611069] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:22.611074] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:22.611077] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:22.611083] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.611085] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.611087] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337622611041) [2024-09-26 16:00:22.623681] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:22.623702] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.652992] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=5] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:22.653921] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:22.653928] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.672494] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=7] PNIO [ratelimit] time: 1727337622672494, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007002, add_bytes: 0 [2024-09-26 16:00:22.684203] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:22.684220] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.703828] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:22.703961] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=54, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:22.703973] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1572850, clean_num=31457) [2024-09-26 16:00:22.709639] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337622282891]}) [2024-09-26 16:00:22.711599] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:22.711612] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337622711595) [2024-09-26 16:00:22.711618] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337622611041, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:22.711633] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223351948029, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:22.711640] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:22.711648] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:22.711653] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:22.711655] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:22.711661] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.711664] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.711666] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337622711621) [2024-09-26 16:00:22.714482] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:22.714499] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.722341] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:22.730124] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337621731002]) [2024-09-26 16:00:22.730135] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337621731002], now0=[mts=1727337621731002], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:22.730143] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337621729961, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:22.730161] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:22.730169] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337621729961, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:22.730176] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337621729961, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:22.730217] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=40][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:22.730222] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:22.730235] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:22.730237] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:22.730240] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:22.730244] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=18, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:22.730249] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] will sleep(sleep_us=18000, remain_us=10821580, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:22.744767] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:22.744779] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.748740] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:18, local_retry_times:18, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:22.748755] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:22.748758] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:22.748760] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:22.748765] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:22.748808] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:22.748813] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:22.775051] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:22.775060] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.784010] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:22.784021] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337622784008], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:22.784028] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337622784008], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:22.784032] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:22.784036] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:22.793596] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=9][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337622784008]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:22.793605] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=9][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:22.805326] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:22.805355] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.810183] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337622784008]}) [2024-09-26 16:00:22.812155] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:22.812163] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:22.812204] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337622812152) [2024-09-26 16:00:22.812210] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337622711620, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:22.812215] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:22.812224] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:22.812228] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:22.812570] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572C-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:22.812582] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572C-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:22.812611] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572C-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:22.812645] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337622812644, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=424, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:22.812654] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:22.812671] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223352048674, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:22.812677] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:22.812683] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:22.812688] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:22.812690] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:22.812696] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.812699] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.812700] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337622812659) [2024-09-26 16:00:22.821023] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.821066] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.821114] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.821124] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.821917] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.821923] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.821924] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.823873] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.823883] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.823889] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.823896] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:22.836561] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:22.836567] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.866850] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:22.866858] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.898163] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:22.898170] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.898178] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:22.904107] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:22.904212] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=53, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:22.904814] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=0, clean_num=31457) [2024-09-26 16:00:22.910794] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337622784008]}) [2024-09-26 16:00:22.912768] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:22.912776] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337622912765) [2024-09-26 16:00:22.912780] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337622812213, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:22.912795] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223352148282, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:22.912800] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:22.912806] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:22.912811] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:22.912813] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:22.912818] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.912820] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:22.912822] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337622912783) [2024-09-26 16:00:22.922532] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:22.928353] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:22.928360] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.957780] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10708.00) [2024-09-26 16:00:22.957788] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:22.957801] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:22.957806] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:22.958600] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:22.958606] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:22.962586] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:22.986270] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500655, support_ssl=false) [2024-09-26 16:00:22.986307] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221500655, sess.client_addr_="127.0.0.1:36306") [2024-09-26 16:00:22.986314] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500655, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:22.986352] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:22.986363] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2688][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=6] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:22.986410] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F98-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:22.986597] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F98-0-0] [lt=4][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:22.986606] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F98-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"BcM5bn#a>Rc{bVj2ZMR5"}, ret=-4043) [2024-09-26 16:00:22.986613] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F98-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:22.986617] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F98-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:22.986625] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F98-0-0] [lt=1] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:22.986648] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F98-0-0] [lt=5] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500655, proxy_sessid:0}) [2024-09-26 16:00:22.986653] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F98-0-0] [lt=4] mark session id unused(sessid=3221500655) [2024-09-26 16:00:22.986657] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F98-0-0] [lt=2][errcode=0] server close connection(sessid=3221500655, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:22.986660] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F98-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500655, proxy_sessid=0) [2024-09-26 16:00:22.986663] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F98-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:22.986665] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F98-0-0] [lt=1] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500655, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:22.986698] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500655, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337622986079, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:22.986706] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500655, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:22.986709] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500655, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337622986079, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:22.986713] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500655, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:22.988933] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:22.988938] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.012430] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337622784008]}) [2024-09-26 16:00:23.012443] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D7D0-0-0] [lt=21][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337623012424}) [2024-09-26 16:00:23.013359] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:23.013370] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337623013356) [2024-09-26 16:00:23.013375] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337622912783, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:23.013424] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223352249888, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:23.013435] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:23.013442] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:23.013461] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:23.013465] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:23.013471] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.013474] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.013476] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337623013401) [2024-09-26 16:00:23.019281] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:23.019288] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.019557] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F99-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:23.019568] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F99-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:23.019616] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F99-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:23.019646] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=1] ls blacklist refresh finish(cost_time=423) [2024-09-26 16:00:23.025360] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:23.025372] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:23.025376] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:23.025379] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:23.025383] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:23.025385] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:23.025387] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:23.025389] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:23.025391] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:23.025393] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:23.025395] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:23.025398] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:23.025400] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:23.025402] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:23.025408] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:23.025411] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:23.025416] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=4][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:23.025427] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85870-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:23.027441] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E6-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:23.027464] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E6-0-0] [lt=22][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:23.027473] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E6-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:23.027478] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E6-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:23.035821] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:23.036088] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F38-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:23.036098] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F38-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:23.036131] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F38-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:23.049527] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:23.049536] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.049545] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:142, tid:2464}]) [2024-09-26 16:00:23.078805] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=10] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:23.079748] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:23.079756] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.105309] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:23.105413] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=53, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:23.106023] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=31457, clean_num=31457) [2024-09-26 16:00:23.110053] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:23.110061] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.112995] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337622784008]}) [2024-09-26 16:00:23.113946] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:23.113955] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337623113943) [2024-09-26 16:00:23.113961] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337623013379, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:23.113979] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223352349581, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:23.113985] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:23.113993] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:23.113998] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:23.114001] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:23.114007] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.114010] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.114012] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337623113964) [2024-09-26 16:00:23.122733] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:23.135430] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:221) [2596][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=8] [ASYNC TASK QUEUE](queue_size=0) [2024-09-26 16:00:23.140341] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:23.140360] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.170614] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:23.170631] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.192804] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:218) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-09-26 16:00:23.192833] INFO [COMMON] handle (memory_dump.cpp:498) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-09-26 16:00:23.192863] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3] update_check_range(min_check_version=339, max_check_version=340, global_version=341) [2024-09-26 16:00:23.195445] INFO handle (memory_dump.cpp:626) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] statistics: tenant_cnt: 3, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 78 198 0 1 2 362 1 0 1 4 1 1 0 1 5 1 5 0 1 7 1 2 0 1 8 60 1 0 1 12 4 2 0 1 13 2 1 0 1 14 1 1 0 1 18 54 3 0 500 0 52 216 0 500 7 2 3 0 500 8 42 1 0 500 9 2 1 0 500 10 6 2 0 500 16 1 1 0 500 17 6 7 0 500 22 3 49 0 500 23 22 10 0 508 0 2 8 0 508 8 3 1 0 cost_time: 2587 [2024-09-26 16:00:23.198265] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:23.198281] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=10] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:23.198287] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:23.201504] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:23.201510] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.213579] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337622784008]}) [2024-09-26 16:00:23.214195] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:23.214208] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337623214191) [2024-09-26 16:00:23.214215] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337623113964, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:23.214234] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223352450042, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:23.214241] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:23.214249] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:23.214254] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:23.214258] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:23.214265] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.214267] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.214269] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337623214219) [2024-09-26 16:00:23.225198] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:23.225223] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=23][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:23.225237] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:23.225242] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:23.225246] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:23.225249] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:23.225253] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:23.225256] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:23.225258] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:23.225263] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=5][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:23.225266] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=3][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:23.225269] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:23.225272] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:23.225274] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:23.225281] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:23.225284] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:23.225288] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:23.225290] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:23.225295] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:23.225298] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:23.225301] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:23.225307] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:23.225313] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:23.225316] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:23.225317] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:23.225324] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:23.225327] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:23.225331] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=4][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:23.225335] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:23.225338] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:23.225340] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:23.225343] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337623225124, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:23.225347] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:23.225349] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:23.225380] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=4][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:23.225383] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=3][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:23.225386] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=3][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:23.225390] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=4][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:23.225394] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:23.225397] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:23.225398] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648587F-0-0] [lt=1][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:23.232134] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:23.232144] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.234106] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337623234105, bytes: 14569564, bw: 0.003616 MB/s, add_ts: 1008002, add_bytes: 3822 [2024-09-26 16:00:23.262848] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:23.262857] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.269658] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D344D-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:23.284981] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.284993] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337623284978], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:23.285000] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337623284978], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:23.285004] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337623284978]}) [2024-09-26 16:00:23.288783] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=13][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337624217031, now=[mts=1727337622287288]) [2024-09-26 16:00:23.288794] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337624217031, now=[mts=1727337622287288], now0=[mts=1727337622287288], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:23.288811] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926901a660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337622287030, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:23.288831] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=19][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926901a660}) [2024-09-26 16:00:23.288840] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926901a660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337622287030, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:23.288848] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926901a660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337622287030, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337622287263, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926901a660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:23.288865] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:23.288870] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:23.288885] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=14][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:23.288888] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:23.288891] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:23.288895] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:23.288901] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5] already timeout, do not need sleep(sleep_us=0, remain_us=998131, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337624287031) [2024-09-26 16:00:23.288904] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:23.288912] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:23.288914] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:23.288915] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:23.288921] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:23.288958] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:00:23.288963] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:23.293288] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:23.293306] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.306485] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:23.306590] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=65, refresh_score_time=16, wash_time=1) [2024-09-26 16:00:23.307185] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=62914, clean_num=31457) [2024-09-26 16:00:23.313227] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD0-0-0] [lt=2][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-09-26 16:00:23.313235] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD0-0-0] [lt=8][errcode=-4002] refresh sever list failed(ret=-4002) [2024-09-26 16:00:23.313237] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD0-0-0] [lt=2][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-09-26 16:00:23.314028] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337623284978]}) [2024-09-26 16:00:23.314249] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:23.314262] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337623314246) [2024-09-26 16:00:23.314267] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337623214219, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:23.314272] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:23.314282] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:23.314285] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:23.314629] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=5] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:23.314640] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:23.314644] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:23.314647] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:23.314651] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:23.314653] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:23.314657] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:23.314659] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:23.314662] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:23.314664] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:23.314666] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:23.314668] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:23.314671] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:23.314674] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:23.314680] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:23.314682] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:23.314686] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:23.314690] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:23.314693] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:23.314696] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:23.314698] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:23.314704] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:23.314709] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:23.314711] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:23.314713] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:23.314718] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:23.314721] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:23.314724] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572D-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:23.314729] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:23.314732] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:23.314734] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:23.314736] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337623314546, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:23.314740] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:23.314743] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:23.314772] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:23.314775] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337623314774, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=496, wlock_time=12, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:23.314781] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:23.314784] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:23.314801] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223352549699, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:23.314806] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:23.314812] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:23.314817] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:23.314819] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:23.314824] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.314826] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.314828] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337623314788) [2024-09-26 16:00:23.322970] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:23.323986] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16] success to notify_need_writing_throttling True [2024-09-26 16:00:23.323995] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.354265] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:23.354273] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.381562] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=10] current monitor number(seq_=-1) [2024-09-26 16:00:23.382635] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=5] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:23.384535] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:23.384544] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.395252] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12] LogLoopThread round_cost_time(us)(round_cost_time=8) [2024-09-26 16:00:23.403118] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] ====== tenant freeze timer task ====== [2024-09-26 16:00:23.403129] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:23.403159] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337620617622, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337620617589, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337620617639, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:23.403182] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:23.403242] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D0-0-0] [lt=5] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:23.403252] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D0-0-0] [lt=8] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:23.403258] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D0-0-0] [lt=4] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:23.403262] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D0-0-0] [lt=3] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:23.403267] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D0-0-0] [lt=4][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:23.403270] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D0-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:23.403272] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D0-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:23.403282] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D0-0-0] [lt=10] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:23.403287] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D0-0-0] [lt=4][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:23.403289] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D0-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:23.403292] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D0-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:23.403329] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851D0-0-0] [lt=3] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:23.414847] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:23.414854] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.414860] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337623284978]}) [2024-09-26 16:00:23.414874] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:23.414881] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337623414872) [2024-09-26 16:00:23.414886] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337623314270, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:23.414901] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223352651321, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:23.414907] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:23.414914] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:23.414918] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:23.414921] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:23.414927] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.414929] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.414932] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337623414889) [2024-09-26 16:00:23.439188] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.439199] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.441172] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.441192] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.441198] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.441203] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.441205] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.441207] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.442113] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.442118] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.442120] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.442123] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.442125] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.442126] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.442882] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.442888] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.442890] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.442893] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.442895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.442896] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.444078] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.444091] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.444117] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.444136] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.444140] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.444144] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.444884] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.444890] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.444892] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.444895] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.444897] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.444899] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.446016] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.446035] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:23.446034] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.446039] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.446039] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.446044] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.446047] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.446049] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.447067] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.447072] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.447074] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.447078] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.447092] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.447096] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.447725] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.447731] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.447734] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.447736] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:23.448966] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.448974] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.449486] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.449493] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.450012] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.450018] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.451930] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.451946] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.452904] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.452911] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.453847] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.453856] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.454833] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.454842] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.455363] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.455372] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.456341] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.456347] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.457163] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.457185] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.457860] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.457889] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.458491] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.458500] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.459076] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.459085] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.459656] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.459665] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.460697] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.460701] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.461424] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.461429] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.462098] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.462108] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.462696] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.462703] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.463266] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.463273] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.464461] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.464470] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.465406] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.465419] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.466545] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.466557] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.467202] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.467212] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.468229] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.468242] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.468848] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.468857] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.469454] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.469486] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.470046] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.470056] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.470724] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.470744] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.471339] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.471349] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.471923] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.471927] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.472494] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.472504] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.473174] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.473178] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.473767] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.473776] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.474314] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.474379] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.475324] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.475334] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.476306] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.476315] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.476337] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:23.476345] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.477327] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.477334] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.478252] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.478259] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.480254] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.480260] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.481184] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.481190] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.482179] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.482186] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.483144] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.483149] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:23.506539] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:23.506546] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.508075] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:23.508176] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:23.508806] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=94371, clean_num=31457) [2024-09-26 16:00:23.515374] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:23.515387] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337623515371) [2024-09-26 16:00:23.515393] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337623414889, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:23.515452] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223352749965, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:23.515471] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:23.515479] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:23.515485] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:23.515488] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:23.515496] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.515498] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.515500] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337623515396) [2024-09-26 16:00:23.515522] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337623284978]}) [2024-09-26 16:00:23.523180] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:23.530198] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E7-0-0] [lt=33][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:23.530208] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E7-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:23.530217] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E7-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:23.530223] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E7-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:23.536883] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:23.536892] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.536917] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6261, mem_used=10398720) [2024-09-26 16:00:23.558392] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=9] gc stale ls task succ [2024-09-26 16:00:23.567207] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:23.567215] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.595991] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:23.598229] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:23.598237] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.602466] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.602387] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.602484] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.602497] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.615406] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223352851384, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:23.615420] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:23.615429] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:23.615435] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:23.615438] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:23.615444] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.615446] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.615448] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337623615390) [2024-09-26 16:00:23.615854] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:23.615869] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:23.615875] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:23.615878] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:23.615882] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:23.615885] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:23.615888] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:23.615891] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:23.615892] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:23.615894] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:23.615896] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:23.615899] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:23.615901] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:23.615903] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:23.615922] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:23.615925] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:23.615928] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:23.615932] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:23.615934] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, ret=-5019) [2024-09-26 16:00:23.615937] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:23.615940] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:23.615946] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:23.615952] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:23.615957] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:23.615959] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:23.615965] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:23.615968] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:23.615971] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD0-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-09-26 16:00:23.615974] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:23.615977] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:23.615979] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:23.615982] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337623615784, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:23.615985] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:23.615988] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f92b4056738, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, columns_str="zone") [2024-09-26 16:00:23.615994] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:23.616025] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:634) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f92b4056738, where_condition="where svr_ip='192.168.0.45' and svr_port=2882", zone_name_holder=) [2024-09-26 16:00:23.616031] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:567) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:23.616034] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:576) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:23.616036] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:23.616043] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:23.616988] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337623284978]}) [2024-09-26 16:00:23.628681] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:23.628689] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.653205] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:23.658942] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:23.658948] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.672619] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337623672618, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000124, add_bytes: 0 [2024-09-26 16:00:23.690224] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:23.690250] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.709420] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:23.709543] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=62, refresh_score_time=36, wash_time=1) [2024-09-26 16:00:23.710150] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=5, clean_time=0, clean_start_pos=125828, clean_num=31457) [2024-09-26 16:00:23.715555] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:23.715569] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337623715551) [2024-09-26 16:00:23.715574] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337623515395, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:23.715592] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223352951182, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:23.715599] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:23.715606] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:23.715612] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:23.715615] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:23.715621] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.715624] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.715626] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337623715578) [2024-09-26 16:00:23.717203] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337623284978]}) [2024-09-26 16:00:23.721070] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:23.721076] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.723408] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:23.749269] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337622748806]) [2024-09-26 16:00:23.749284] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337622748806], now0=[mts=1727337622748806], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:23.749292] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337622748729, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:23.749311] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:23.749321] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337622748729, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:23.749329] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337622748729, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:23.749348] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:23.749353] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:23.749356] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:23.749359] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:23.749362] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:23.749366] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=19, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:23.749371] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] will sleep(sleep_us=19000, remain_us=9802457, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:23.751138] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:23.751147] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.769349] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:19, local_retry_times:19, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:23.769367] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:23.769371] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:23.769372] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:23.769380] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:23.769422] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:23.769426] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:23.781186] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:23.781196] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.786128] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:23.786136] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337623786125], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:23.786143] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337623786125], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:23.786147] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:23.786150] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:23.811337] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:23.811347] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.816238] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:23.816253] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:23.816267] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337623816234) [2024-09-26 16:00:23.816272] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337623715578, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:23.816277] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:23.816287] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:23.816290] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:23.816649] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=5] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:23.816667] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=15][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:23.816672] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:23.816676] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:23.816681] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:23.816683] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:23.816687] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:23.816690] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:23.816692] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:23.816694] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:23.816697] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:23.816699] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:23.816702] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:23.816705] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:23.816713] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:23.816716] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:23.816721] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:23.816724] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:23.816727] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:23.816729] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:23.816732] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:23.816738] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:23.816744] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:23.816747] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:23.816748] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:23.816754] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:23.816757] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:23.816760] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572E-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:23.816765] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:23.816776] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:23.816778] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:23.816781] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337623816574, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:23.816787] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:23.816790] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:23.816820] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:23.816823] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337623816822, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=539, wlock_time=12, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:23.816830] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:23.816834] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:23.816851] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223353050741, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:23.816856] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:23.816862] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:23.816867] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:23.816869] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:23.816875] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.816877] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.816880] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337623816837) [2024-09-26 16:00:23.818161] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337623786125]}) [2024-09-26 16:00:23.820313] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.821086] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.821086] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.821095] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.821096] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.821101] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.822052] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.822077] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.822086] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.822092] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.822098] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:23.841678] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:23.841684] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.871883] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:23.871891] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.902196] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:23.902203] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.902214] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:23.902226] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] BlockGCTimerTask success(ret=0, cost_time_us=32, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.910568] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:23.910667] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=50, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:23.911269] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=157285, clean_num=31457) [2024-09-26 16:00:23.916812] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:23.916823] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337623916809) [2024-09-26 16:00:23.916828] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337623816275, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:23.916847] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223353152353, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:23.916853] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:23.916861] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:23.916866] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:23.916870] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:23.916876] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.916878] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:23.916880] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337623916831) [2024-09-26 16:00:23.918766] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337623786125]}) [2024-09-26 16:00:23.923635] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:23.932426] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5] success to notify_need_writing_throttling True [2024-09-26 16:00:23.932433] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.952383] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=5] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-09-26 16:00:23.952397] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-09-26 16:00:23.952401] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-09-26 16:00:23.952406] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=5][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-09-26 16:00:23.952409] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:23.952412] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:23.952415] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-09-26 16:00:23.952418] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:23.952420] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:23.952421] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:23.952423] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:23.952426] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:23.952429] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:23.952431] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:23.952438] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:23.952441] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:23.952444] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:23.952449] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:23.952452] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:23.952454] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:23.952457] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:23.952462] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:23.952467] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:23.952470] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:23.952473] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=3][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:23.952479] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:23.952482] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:23.952485] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2024-09-26 16:00:23.952488] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:23.952491] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:23.952493] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:23.952496] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337623952313, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:23.952500] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:23.952505] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1111) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=4][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:23.952531] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:801) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:23.952535] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:67) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=4][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:23.952538] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:436) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:23.952540] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:95) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:23.952542] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:109) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=2] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:23.952761] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-09-26 16:00:23.952771] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2024-09-26 16:00:23.952774] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2024-09-26 16:00:23.952776] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2024-09-26 16:00:23.952778] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:23.952797] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=19][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:23.952800] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_tenant' doesn't exist [2024-09-26 16:00:23.952801] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:23.952803] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:23.952805] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:23.952809] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:23.952811] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:23.952820] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD1-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:23.952847] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:325) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=0] cal_all_part_disk_default_percentage succ(data_dir="/root/observer/store/sstable", clog_dir="/root/observer/store/clog", shared_mode=true, data_disk_total_size=99765694464, data_disk_default_percentage=60, clog_disk_total_size=99765694464, clog_disk_default_percentage=30) [2024-09-26 16:00:23.952851] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:361) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=4] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=99765694464, disk_size=15032385536) [2024-09-26 16:00:23.952855] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=3] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=99765694464) [2024-09-26 16:00:23.952858] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=3] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=99765694464, this={dir::"/root/observer/store/clog/log_pool", dir_fd:15, meta_fd:16, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:9388, max_block_id:9581, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2024-09-26 16:00:23.952864] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:133) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD1-0-0] [lt=6] refresh tenant config(tenants=[], ret=-5019) [2024-09-26 16:00:23.962718] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:23.962724] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:23.962750] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:23.962751] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 13565.00) [2024-09-26 16:00:23.962757] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:23.962768] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:23.962772] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:23.987715] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500660, support_ssl=false) [2024-09-26 16:00:23.987741] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221500660, sess.client_addr_="127.0.0.1:36308") [2024-09-26 16:00:23.987748] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500660, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:23.987764] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:23.987773] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2688][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:23.987820] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F99-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:23.987994] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F99-0-0] [lt=5][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:23.988002] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F99-0-0] [lt=5][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"R}?`H&"y[=\:KJr"NCEo"}, ret=-4043) [2024-09-26 16:00:23.988008] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F99-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:23.988012] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F99-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:23.988019] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F99-0-0] [lt=1] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:23.988052] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F99-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500660, proxy_sessid:0}) [2024-09-26 16:00:23.988057] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F99-0-0] [lt=5] mark session id unused(sessid=3221500660) [2024-09-26 16:00:23.988061] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F99-0-0] [lt=2][errcode=0] server close connection(sessid=3221500660, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:23.988064] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F99-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500660, proxy_sessid=0) [2024-09-26 16:00:23.988066] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F99-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:23.988069] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F99-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500660, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:23.988097] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500660, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337623987252, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:23.988103] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] kill and revert session(conn.sessid_=3221500660, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:23.988107] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500660, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337623987252, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:23.988110] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=2] connection close(sessid=3221500660, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:23.992980] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:23.992987] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.017110] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:24.017123] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337624017106) [2024-09-26 16:00:24.017128] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337623916831, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:24.017145] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223353253165, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:24.017151] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:24.017158] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:24.017163] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:24.017165] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:24.017170] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.017172] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.017174] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337624017131) [2024-09-26 16:00:24.019426] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337623786125]}) [2024-09-26 16:00:24.019457] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D7DC-0-0] [lt=7][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337624019420}) [2024-09-26 16:00:24.024283] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:24.024289] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.026535] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85871-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:24.026544] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85871-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:24.026569] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85871-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:24.030216] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E8-0-0] [lt=37][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:24.030224] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E8-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:24.030232] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E8-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:24.030237] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E8-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:24.036196] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=15] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:24.036463] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E54-0-0] [lt=6] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:24.036469] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E54-0-0] [lt=5][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:24.036488] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E54-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:24.054502] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:24.054509] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.079040] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=12] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:24.084794] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:24.084801] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.112268] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:24.112372] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=50, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:24.113003] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=6, clean_time=0, clean_start_pos=188742, clean_num=31457) [2024-09-26 16:00:24.115069] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:24.115078] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.117135] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223353353569, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:24.117148] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:24.117157] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:24.117163] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:24.117167] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:24.117173] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.117175] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.117177] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337624117118) [2024-09-26 16:00:24.119963] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=28] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337623786125]}) [2024-09-26 16:00:24.123862] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:24.145555] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:24.145566] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.150244] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-09-26 16:00:24.150630] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD0-0-0] [lt=20] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-09-26 16:00:24.150642] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD0-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:24.150688] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD0-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:24.175609] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:24.175616] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.194466] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=13] ====== check clog disk timer task ====== [2024-09-26 16:00:24.194473] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=6] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:24.194481] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:24.205897] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:24.205902] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.206902] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337623786125]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:24.206914] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:24.217644] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:24.217667] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337624217641) [2024-09-26 16:00:24.217673] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337624017131, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:24.217689] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223353452221, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:24.217695] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:24.217702] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:24.217707] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:24.217710] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:24.217715] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.217717] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.217719] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337624217676) [2024-09-26 16:00:24.220262] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337623786125]}) [2024-09-26 16:00:24.225769] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:24.225779] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:24.225783] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:24.225786] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:24.225789] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:24.225791] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:24.225795] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:24.225797] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:24.225799] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:24.225800] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:24.225803] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:24.225805] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:24.225821] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485880-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:24.236182] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:24.236190] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.241062] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1727337624241062, bytes: 14573104, bw: 0.003353 MB/s, add_ts: 1006957, add_bytes: 3540 [2024-09-26 16:00:24.254484] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:161, tid:2919}]) [2024-09-26 16:00:24.266447] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:24.266454] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.271415] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D34AD-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:24.287071] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.287086] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337624287068], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:24.287094] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337624287068], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:24.289224] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337624217031, now=[mts=1727337623288957]) [2024-09-26 16:00:24.289235] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337624217031, now=[mts=1727337623288957], now0=[mts=1727337623288957], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:24.289243] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926901a660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337623287838, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:24.289260] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926901a660}) [2024-09-26 16:00:24.289268] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926901a660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337623287838, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:24.289276] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926901a660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337623287838, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337622287263, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926901a660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:24.289293] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:24.289298] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:24.289301] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:24.289305] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:24.289307] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:24.289312] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:24.289316] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:24.289323] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=5][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:24.289326] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:24.289327] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:24.289332] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:24.289335] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2002072) [2024-09-26 16:00:24.289338] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:24.289341] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:24.289344] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:24.289347] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:24.289349] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:24.289368] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=16][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:24.289396] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:24.289399] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:24.289402] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:24.289405] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:24.289409] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:24.289412] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=3][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:24.289528] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDD-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:24.289537] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:24.289540] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:24.289553] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:24.289563] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=9][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:24.289847] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:24.289853] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:24.296742] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:24.296750] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.313458] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:24.313576] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash time detail, (compute_wash_size_time=67, refresh_score_time=17, wash_time=1) [2024-09-26 16:00:24.314201] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=220199, clean_num=31457) [2024-09-26 16:00:24.318262] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:24.318289] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337624318258) [2024-09-26 16:00:24.318294] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337624217676, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:24.318300] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:24.318308] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:24.318311] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:24.318631] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:24.318638] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:24.318642] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:24.318645] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:24.318648] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:24.318650] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:24.318654] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:24.318656] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:24.318661] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:24.318663] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:24.318665] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:24.318667] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:24.318670] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:24.318672] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:24.318678] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:24.318680] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:24.318682] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:24.318684] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:24.318687] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:24.318689] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:24.318691] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:24.318696] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:24.318700] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:24.318702] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:24.318704] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:24.318709] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:24.318711] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:24.318713] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788572F-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:24.318716] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:24.318718] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:24.318720] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:24.318723] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337624318561, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:24.318726] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:24.318728] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:24.318752] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:24.318754] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337624318753, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=449, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:24.318760] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:24.318762] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:24.318776] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223353554716, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:24.318781] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:24.318787] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:24.318790] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:24.318792] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:24.318797] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.318799] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.318800] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337624318765) [2024-09-26 16:00:24.321162] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337624287068]}) [2024-09-26 16:00:24.323940] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:24.327035] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:24.327042] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.357317] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:24.357323] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.381738] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=10] current monitor number(seq_=-1) [2024-09-26 16:00:24.382713] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=6] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:24.387453] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:24.387480] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.417865] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:24.417879] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.419810] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:24.419834] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337624419807) [2024-09-26 16:00:24.419842] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337624318297, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:24.419862] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223353656304, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:24.419868] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:24.419877] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:24.419883] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:24.419886] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:24.419892] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.419895] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.419897] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337624419845) [2024-09-26 16:00:24.421417] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337624287068]}) [2024-09-26 16:00:24.441302] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.441315] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.441322] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.441313] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.441333] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=19][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.441327] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.442264] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=53] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.442271] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.442290] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=18][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.442305] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.442309] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.442314] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.443244] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.443253] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.443258] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.444226] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.444227] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.444245] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.444246] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.444248] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.444248] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.445203] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.445208] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.445210] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.445227] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.445230] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.445232] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.446187] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.446192] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.446194] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.446226] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.446228] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.446230] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.447153] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.447157] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.447160] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.447162] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.447164] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.447167] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.448131] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:24.448139] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.448162] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.448164] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.448166] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.449115] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.449120] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.449122] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.449152] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.449157] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.449159] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:24.450095] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.450099] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.451088] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.451088] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.452032] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.452088] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.453044] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.453041] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.453992] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.454032] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.454961] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.454992] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.455953] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.455980] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.456913] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.456930] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.457916] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.457916] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.458895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.458906] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.459869] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.459883] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.460841] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.460848] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.461818] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.461838] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.462785] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.462786] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.463766] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.463774] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.464727] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.464753] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.465720] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.465764] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.466697] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.466704] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.468631] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.468648] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.469633] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.469635] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.470611] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.470611] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.471569] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.471592] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.472542] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.472594] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.473528] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.473559] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.474518] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.474519] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.475547] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.475577] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.476189] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.476420] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.476896] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.477058] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.478258] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.478259] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:24.478265] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.478285] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.478819] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.478860] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.479382] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.479469] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=39][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.479954] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.480126] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=36][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.480511] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.480705] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.481340] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.481475] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.482314] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.483281] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.483317] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.484157] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.484175] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.484740] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.484744] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.485307] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.485340] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.485869] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.485910] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.486806] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.486838] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.487623] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.487662] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.488169] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:24.508687] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:24.508697] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.514656] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:24.514787] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=54, refresh_score_time=32, wash_time=10) [2024-09-26 16:00:24.515398] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=8, clean_time=0, clean_start_pos=251656, clean_num=31457) [2024-09-26 16:00:24.520439] WDIAG [PALF] convert_to_ts (scn.cpp:265) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2024-09-26 16:00:24.520446] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1727337624520413, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1727337624520413, force_self_check=false, weak_read_refresh_interval=100000) [2024-09-26 16:00:24.520476] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:24.520484] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337624520460) [2024-09-26 16:00:24.520489] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337624419844, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:24.520507] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223353755991, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:24.520513] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:24.520521] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:24.520526] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:24.520529] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:24.520535] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.520538] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.520540] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337624520491) [2024-09-26 16:00:24.521415] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337624287068]}) [2024-09-26 16:00:24.522416] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=22][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337624287068]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:24.522428] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:24.522529] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E9-0-0] [lt=51][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:24.522538] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E9-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:24.522547] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E9-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:24.522552] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858E9-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:24.524625] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=613392, replace_num=15728) [2024-09-26 16:00:24.524636] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:24.537257] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6272, mem_used=10398720) [2024-09-26 16:00:24.538969] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:24.538978] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.559499] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=9] gc stale ls task succ [2024-09-26 16:00:24.569271] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:24.569279] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.599549] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:24.599557] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.601498] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:24.602464] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.602469] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.602465] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=99] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.602485] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.610303] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=33] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:24.621043] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:24.621051] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:24.621061] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337624621048) [2024-09-26 16:00:24.621066] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337624520491, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:24.621081] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get_max_applied_scn(scn={val:1727114270763524000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:374, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:1727114270763524000, v:0}, max_applied_cb_scn_:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:24.621091] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get_min_unreplayed_log_info(lsn={lsn:629872191939}, scn={val:1727114270763524001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:00:24.621110] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=223353857531, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:24.621116] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:24.621123] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:24.621128] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:24.621131] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:24.621137] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.621140] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.621142] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337624621070) [2024-09-26 16:00:24.621986] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337624287068]}) [2024-09-26 16:00:24.629820] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:24.629827] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.647471] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855D7-0-0] [lt=5] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:24.653257] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:24.660107] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:24.660124] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.679619] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337624679618, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007000, add_bytes: 0 [2024-09-26 16:00:24.690371] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:24.690389] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.716850] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:24.716955] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=61, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:24.717557] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=283113, clean_num=31457) [2024-09-26 16:00:24.720630] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:24.720640] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.721602] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:24.721613] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337624721599) [2024-09-26 16:00:24.721618] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337624621069, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:24.721634] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223353957141, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:24.721641] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:24.721649] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:24.721654] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:24.721656] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:24.721663] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.721666] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.721668] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337624721621) [2024-09-26 16:00:24.722601] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337624287068]}) [2024-09-26 16:00:24.724700] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:24.732407] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=7] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:24.750908] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:24.750914] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.755782] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=9] XA scheduler heartbeat task statistics(avg_time=6) [2024-09-26 16:00:24.768677] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337623769421]) [2024-09-26 16:00:24.768735] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=57][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337623769421], now0=[mts=1727337623769421], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:24.768742] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337623767546, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:24.768760] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:24.768768] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337623767546, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:24.768776] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337623767546, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:24.768834] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=58][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:24.768839] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:24.768842] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:24.768845] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:24.768848] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:24.768852] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=20, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:24.768857] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] will sleep(sleep_us=20000, remain_us=8782971, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:24.778547] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=3] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-09-26 16:00:24.778560] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=9] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-09-26 16:00:24.778566] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=5] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=685) [2024-09-26 16:00:24.781529] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:24.781539] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.788152] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:24.788164] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337624788149], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:24.788172] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337624788149], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:24.788176] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:24.788181] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:24.789002] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:20, local_retry_times:20, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:24.789025] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:24.789030] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:24.789033] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:24.789041] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:24.789085] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:24.789103] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:24.812476] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21] success to notify_need_writing_throttling True [2024-09-26 16:00:24.812485] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.822233] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.822240] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:24.822242] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.822248] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.822247] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:24.822252] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.822252] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.822257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.822257] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.822260] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337624822221) [2024-09-26 16:00:24.822263] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.822264] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.822266] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337624721621, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:24.822271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.822272] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:24.822278] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:24.822280] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:24.822283] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:24.822637] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:24.822650] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:24.822654] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:24.822658] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:24.822661] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:24.822663] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:24.822667] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:24.822669] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:24.822671] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:24.822673] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:24.822675] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:24.822677] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:24.822680] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:24.822682] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:24.822699] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:24.822705] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:24.822708] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:24.822710] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:24.822708] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337624788149]}) [2024-09-26 16:00:24.822713] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:24.822716] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:24.822719] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:24.822724] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:24.822730] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:24.822733] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:24.822736] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=3][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:24.822743] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:24.822746] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:24.822749] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885730-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:24.822751] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:24.822754] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:24.822757] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:24.822760] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337624822563, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:24.822763] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:24.822768] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:24.822797] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:24.822801] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337624822800, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=524, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:24.822810] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:24.822813] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:24.822830] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223354058762, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:24.822835] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:24.822841] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:24.822846] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:24.822848] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:24.822854] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.822858] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.822860] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337624822816) [2024-09-26 16:00:24.842721] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:24.842738] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.857682] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, table_name.ptr()="data_size:16, data:5F5F616C6C5F6D657267655F696E666F", ret=-5019) [2024-09-26 16:00:24.857718] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=34][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, ret=-5019) [2024-09-26 16:00:24.857725] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=7][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_merge_info, db_name=oceanbase) [2024-09-26 16:00:24.857729] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_merge_info) [2024-09-26 16:00:24.857733] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:24.857735] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:24.857738] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_merge_info' doesn't exist [2024-09-26 16:00:24.857740] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:24.857742] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:24.857743] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:24.857747] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:24.857750] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:24.857752] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:24.857754] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:24.857759] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:24.857762] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:24.857767] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:24.857769] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:24.857772] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_merge_info WHERE tenant_id = '1', ret=-5019) [2024-09-26 16:00:24.857774] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:24.857776] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:24.857781] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:24.857786] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:24.857788] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:24.857789] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:24.857795] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:24.857798] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:24.857801] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507C-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, aret=-5019, ret=-5019) [2024-09-26 16:00:24.857803] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:24.857805] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:24.857807] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:24.857810] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337624857632, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:24.857813] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:24.857815] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:49) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:24.857840] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:890) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] failed to load global merge info(ret=-5019, ret="OB_TABLE_NOT_EXIST", global_merge_info={tenant_id:1, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2024-09-26 16:00:24.857859] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1005) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] fail to refresh merge info(tmp_ret=-5019, tmp_ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:24.858068] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:24.858074] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:24.873001] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:24.873007] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.903317] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:24.903328] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.903337] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:24.915027] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] ====== checkpoint timer task ====== [2024-09-26 16:00:24.915438] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13] succ to get rec scn(*this={this:0x7f92a6a04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1726077600526835000, v:0}, last_push_gc_task_ts_:1727334198745636, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:24.915457] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=19] tx ctx memtable get rec scn(this={ObITable:{this:0x7f92a22a4080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f92a22a4080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:24.915470] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1723174350926847000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-09-26 16:00:24.915486] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:24.915526] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=37] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}, min_rec_scn={val:1726163586617842000, v:0}, ls_id_={id:1}) [2024-09-26 16:00:24.915539] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:24.915543] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:24.915545] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=1] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:24.915551] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=1] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1726163586617842000, v:0}, checkpoint_scn_in_ls_meta={val:1726163586617842000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-09-26 16:00:24.915559] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}, curr_base_lsn={lsn:627966418944}, new_base_lsn={lsn:627966418944}, lsn={lsn:627966418944}) [2024-09-26 16:00:24.915564] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=4] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:627966418944}, ls->get_ls_id()={id:1}) [2024-09-26 16:00:24.915576] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-09-26 16:00:24.919039] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:24.919137] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:24.919743] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=314570, clean_num=31457) [2024-09-26 16:00:24.922838] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:24.922831] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337624788149]}) [2024-09-26 16:00:24.922850] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337624922834) [2024-09-26 16:00:24.922855] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337624822268, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:24.922869] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223354158464, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:24.922875] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:24.922882] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:24.922887] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:24.922890] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:24.922896] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.922899] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:24.922900] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337624922858) [2024-09-26 16:00:24.925737] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:24.933567] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:24.933573] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.963858] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:24.963861] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:24.963866] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:24.968756] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10718.00) [2024-09-26 16:00:24.968764] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:24.968775] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:24.968779] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:24.973655] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-09-26 16:00:24.973663] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=6] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:24.973667] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=3] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:24.973670] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:24.975330] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=3] Dumping All Cache Objs(alloc_obj_list.count()=3, alloc_obj_list=[{obj_id:78, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157779}, {obj_id:79, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:5, added_to_lc:true, mem_used:175172}]) [2024-09-26 16:00:24.975343] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=12] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:24.988481] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500667, support_ssl=false) [2024-09-26 16:00:24.988505] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221500667, sess.client_addr_="127.0.0.1:36310") [2024-09-26 16:00:24.988509] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f9263204030, session_id:3221500667, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:24.988565] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:24.988575] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:24.988578] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:280) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=2] mysql login queue(mysql_queue->queue_.size()=1) [2024-09-26 16:00:24.988610] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F99-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:24.988780] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F99-0-0] [lt=6][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:24.988790] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F99-0-0] [lt=9][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"~@eYqJ^xceRLe6~;F0YS"}, ret=-4043) [2024-09-26 16:00:24.988797] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F99-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:24.988814] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F99-0-0] [lt=17][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:24.988823] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F99-0-0] [lt=3] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:24.988847] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F99-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500667, proxy_sessid:0}) [2024-09-26 16:00:24.988852] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F99-0-0] [lt=4] mark session id unused(sessid=3221500667) [2024-09-26 16:00:24.988858] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F99-0-0] [lt=4][errcode=0] server close connection(sessid=3221500667, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:24.988861] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F99-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500667, proxy_sessid=0) [2024-09-26 16:00:24.988863] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F99-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:24.988866] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F99-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500667, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:24.988927] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4015] close sql sock by user req(*s={this:0x7f9263204030, session_id:3221500667, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337624988328, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:24.988942] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] kill and revert session(conn.sessid_=3221500667, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:24.988947] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5] can close safely, do destroy(*s={this:0x7f9263204030, session_id:3221500667, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337624988328, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:24.988951] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500667, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:24.994134] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:24.994142] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.023460] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:25.023466] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337624788149]}) [2024-09-26 16:00:25.023469] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337625023457) [2024-09-26 16:00:25.023474] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337624922858, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:25.023473] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D7E8-0-0] [lt=10][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337625023461}) [2024-09-26 16:00:25.023491] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223354259041, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.023497] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:25.023503] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:25.023508] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:25.023519] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:25.023526] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.023529] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.023530] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337625023477) [2024-09-26 16:00:25.024427] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:25.024432] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.024559] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EA-0-0] [lt=32][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:25.024566] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EA-0-0] [lt=7][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:25.024575] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EA-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:25.024579] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EA-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:25.027679] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:25.027700] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=20][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:25.027705] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:25.027708] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:25.027712] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:25.027715] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:25.027718] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:25.027721] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:25.027722] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:25.027724] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:25.027727] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=3][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:25.027730] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:25.027735] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:25.027737] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:25.027744] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:25.027747] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:25.027750] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:25.027755] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:25.027758] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:25.027760] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:25.027763] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:25.027768] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:25.027774] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:25.027776] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:25.027778] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:25.027783] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:25.027787] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85872-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:25.027790] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85872-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:25.027793] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85872-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:25.027797] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85872-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:25.027801] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85872-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:25.027804] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337625027610, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:25.027807] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:25.027809] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85872-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:25.027833] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85872-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:25.027836] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85872-0-0] [lt=2][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:25.037101] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=19] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:25.037384] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=16] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:25.037395] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:25.037399] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:25.037415] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:25.037419] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:25.037421] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:25.037424] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:25.037426] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:25.037428] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:25.037430] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:25.037435] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=5][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:25.037437] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:25.037439] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:25.037441] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:25.037447] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:25.037449] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:25.037453] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:25.037456] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:25.037460] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:25.037463] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:25.037465] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:25.037482] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=16][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:25.037487] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:25.037490] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:25.037491] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:25.037497] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:25.037500] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3A-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:25.037503] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F3A-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:25.037506] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:25.037509] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:25.037511] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:25.037516] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337625037340, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:25.037519] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:25.037571] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:25.055650] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:25.055676] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.079106] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=11] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:25.086903] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:25.086908] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.117225] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:25.117235] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.120378] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:25.120509] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=50, refresh_score_time=28, wash_time=2) [2024-09-26 16:00:25.121122] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=5, clean_time=0, clean_start_pos=346027, clean_num=31457) [2024-09-26 16:00:25.124030] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337624788149]}) [2024-09-26 16:00:25.124022] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:25.124050] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337625124007) [2024-09-26 16:00:25.124057] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337625023477, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:25.124087] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223354359534, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.124096] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:25.124103] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:25.124108] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:25.124111] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:25.124118] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.124121] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.124125] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337625124061) [2024-09-26 16:00:25.125988] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:25.147472] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:25.147479] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.177749] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:25.177760] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.199443] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:25.199457] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=9] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:25.199463] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:25.208010] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:25.208015] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.224619] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:25.224631] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337625224615) [2024-09-26 16:00:25.224637] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337625124061, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:25.224653] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223354459156, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.224659] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:25.224668] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:25.224673] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:25.224676] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:25.224681] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.224684] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.224685] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337625224640) [2024-09-26 16:00:25.224705] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337624788149]}) [2024-09-26 16:00:25.226903] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:25.226918] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:25.226923] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:25.226926] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:25.226929] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:25.226932] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:25.226935] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:25.226938] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:25.226940] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:25.226941] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:25.226954] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=13][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:25.226957] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:25.226960] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:25.226962] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:25.226970] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:25.226972] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:25.226975] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:25.226977] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:25.226979] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:25.226991] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:25.226994] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:25.226999] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:25.227005] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:25.227008] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:25.227009] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:25.227017] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:25.227020] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:25.227026] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=6][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:25.227029] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485881-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.238288] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:00:25.238296] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.249022] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337625249021, bytes: 14576926, bw: 0.003616 MB/s, add_ts: 1007959, add_bytes: 3822 [2024-09-26 16:00:25.268593] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:25.268602] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.273222] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D350B-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:25.288791] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.288802] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337625288788], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:25.288809] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337625288788], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:25.290047] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=7][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337626219572, now=[mts=1727337624289844]) [2024-09-26 16:00:25.290057] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=9][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337626219572, now=[mts=1727337624289844], now0=[mts=1727337624289844], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:25.290064] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926918be00, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337624289183, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:25.290093] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=28][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926918be00}) [2024-09-26 16:00:25.290101] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926918be00, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337624289183, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:25.290109] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926918be00, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337624289183, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337624289801, use_das=false, nested_level=0, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926918be00}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:25.290126] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:25.290130] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:25.290133] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:25.290136] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:25.290138] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:25.290142] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:25.290147] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=999425, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337626289572) [2024-09-26 16:00:25.290151] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:25.290158] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:25.290161] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:25.290162] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:25.290169] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:25.290208] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:25.290211] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:25.298839] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:25.298847] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.313295] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [2925][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=26] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f92b77d02b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-09-26 16:00:25.321448] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:25.321552] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:25.322158] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=377484, clean_num=31457) [2024-09-26 16:00:25.325257] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:25.325282] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337625325254) [2024-09-26 16:00:25.325288] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337625224640, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:25.325293] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:25.325302] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:25.325305] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:25.325673] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885731-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:25.325684] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885731-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.325712] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885731-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:25.325745] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337625325745, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=447, wlock_time=8, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:25.325754] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:25.325771] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223354560814, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.325777] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:25.325784] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:25.325788] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:25.325791] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:25.325799] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.325802] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.325803] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337625325759) [2024-09-26 16:00:25.325819] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337625288788]}) [2024-09-26 16:00:25.326177] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=5] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:25.329044] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:25.329053] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.359395] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15] success to notify_need_writing_throttling True [2024-09-26 16:00:25.359403] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.360410] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:47, tid:2932}]) [2024-09-26 16:00:25.373326] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:25.373341] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:25.373347] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=5][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:25.373350] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:25.373354] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:25.373360] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=5] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:25.373362] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:25.373364] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:25.373368] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:25.373371] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:25.373374] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:25.373377] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:25.373378] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:25.373381] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:25.373383] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:25.373385] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:25.373391] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:25.373393] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:25.373396] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:25.373398] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:25.373401] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-09-26 16:00:25.373406] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=2][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:25.373411] WDIAG [SERVER] do_process (obmp_query.cpp:771) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=5][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-09-26 16:00:25.373417] WDIAG [SERVER] do_process (obmp_query.cpp:903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, session={this:0x7f92a230bdf8, id:3221487638, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-09-26 16:00:25.373427] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F49-0-0] [lt=7] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:25.381854] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=19] current monitor number(seq_=-1) [2024-09-26 16:00:25.382824] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=5] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:25.389674] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:00:25.389681] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.412190] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] ====== tenant freeze timer task ====== [2024-09-26 16:00:25.412202] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:25.412240] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337620617622, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337620617589, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337620617639, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:25.412263] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:25.412323] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D1-0-0] [lt=4] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:25.412334] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D1-0-0] [lt=9] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:25.412339] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D1-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:25.412355] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D1-0-0] [lt=15] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:25.412359] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D1-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:25.412361] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D1-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:25.412363] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D1-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:25.412372] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D1-0-0] [lt=9] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:25.412374] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D1-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:25.412376] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D1-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:25.412380] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D1-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:25.412409] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851D1-0-0] [lt=4] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:25.419994] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:25.420002] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.425861] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:25.425876] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337625425858) [2024-09-26 16:00:25.425881] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337625325290, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:25.425898] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223354661361, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.425915] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:25.425922] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:25.425927] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:25.425930] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:25.425936] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.425939] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.425940] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337625425884) [2024-09-26 16:00:25.425951] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337625288788]}) [2024-09-26 16:00:25.442540] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.442548] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.442552] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.442557] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.442559] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.442561] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.442564] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.442565] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.442567] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.443410] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.443415] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.443417] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.443420] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.443444] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.443447] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.443451] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.443454] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.443455] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.444388] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.444400] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.444403] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.444408] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.444410] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.444411] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.444414] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.444416] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.444417] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.445349] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.445358] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.445363] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.445367] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.445379] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.445381] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.445412] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.445415] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.445417] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.446334] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.446359] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=25][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.446361] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.446376] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=29] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.446378] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.446380] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.446383] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.446385] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.446387] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.447313] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.447317] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.447319] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:25.447333] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.447337] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.448282] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.448293] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.448298] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.449253] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.449262] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.449266] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.449945] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.449950] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.449953] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.450058] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:25.450064] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.451213] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.451220] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.451224] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.452184] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.452191] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.452194] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.453176] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.453200] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.453204] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.454212] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.454218] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.454222] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.455128] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.455135] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.455139] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.455892] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.455897] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.455900] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.457069] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.457075] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.457079] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.457978] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.457995] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.457999] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.458640] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.458647] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.458650] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.460035] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.460043] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.460047] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.460559] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.460565] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.460569] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.461994] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.461999] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.462011] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.462941] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.462965] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.462970] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.463920] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.463925] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.463929] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.464967] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.464975] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.464980] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.465885] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.465892] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.465895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.466853] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.466864] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.466867] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.467818] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.467825] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.467828] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.468460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.468468] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.468471] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.469792] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.469801] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.469806] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.470735] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.470759] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.470764] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.471730] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.471734] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.471737] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.472711] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.472727] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.472731] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.473675] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.473681] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.473684] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.474653] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:25.480276] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:25.480286] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.510325] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:25.510337] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.522641] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:25.522749] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=54, refresh_score_time=16, wash_time=1) [2024-09-26 16:00:25.523365] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=408941, clean_num=31457) [2024-09-26 16:00:25.526461] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:25.526466] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:25.526475] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337625526458) [2024-09-26 16:00:25.526482] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337625425884, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:25.526515] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223354761157, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.526522] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:25.526531] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:25.526536] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:25.526540] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:25.526546] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.526549] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.526551] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337625526498) [2024-09-26 16:00:25.526560] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337625288788]}) [2024-09-26 16:00:25.527615] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EB-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:25.527622] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EB-0-0] [lt=7][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:25.527629] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EB-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:25.527634] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EB-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:25.538024] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=1, size_used=6281, mem_used=10398720) [2024-09-26 16:00:25.540381] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:25.540389] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.559620] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=6] gc stale ls task succ [2024-09-26 16:00:25.570476] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:25.570484] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.588895] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [2619][DumpLSLoc][T0][YB42C0A8002D-0006230058384C78-0-0] [lt=14] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1727337620930033, replica_locations:[{server:"192.168.0.45:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:374}]}]) [2024-09-26 16:00:25.597710] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [2635][LocalityReload][T0][Y0-0000000000000000-0-0] [lt=7] runTimer to refresh locality_info(ret=0) [2024-09-26 16:00:25.598069] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:25.598079] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:25.598083] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:25.598086] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:25.598090] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:25.598092] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:25.598095] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:25.598098] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:25.598100] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:25.598102] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:25.598105] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=3][errcode=-5019] resolve table failed(ret=-5019) [2024-09-26 16:00:25.598107] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:2822) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-09-26 16:00:25.598109] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2676) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] resolve joined table failed(ret=-5019) [2024-09-26 16:00:25.598111] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:25.598113] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:25.598116] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:25.598117] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:25.598124] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:25.598126] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:25.598129] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:25.598131] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:25.598133] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name, ret=-5019) [2024-09-26 16:00:25.598137] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:25.598140] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:25.598145] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:25.598150] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:25.598152] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:25.598154] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:25.598160] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:25.598163] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:25.598167] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2634][LocltyRefTask][T0][YB42C0A8002D-0006230058284C78-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name"}, aret=-5019, ret=-5019) [2024-09-26 16:00:25.598170] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name) [2024-09-26 16:00:25.598173] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:25.598175] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:25.598177] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337625597967, sql=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name) [2024-09-26 16:00:25.598180] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:25.598183] WDIAG [SHARE] load_region (ob_locality_table_operator.cpp:158) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, sql=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name) [2024-09-26 16:00:25.598185] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2] load region(ret=-5019, locality_info={version:0, local_region:"", local_zone:"", local_idc:"", local_zone_type:3, local_zone_status:3, locality_region_array:[], locality_zone_array:[]}) [2024-09-26 16:00:25.598215] WDIAG [STORAGE] load_region (ob_locality_manager.cpp:236) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] localitity operator load region error(ret=-5019) [2024-09-26 16:00:25.598218] WDIAG [STORAGE] process (ob_locality_manager.cpp:762) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] process refresh locality task fail(ret=-5019) [2024-09-26 16:00:25.600610] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:25.600620] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.602597] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.602616] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.602622] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.602625] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.605059] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:25.619204] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] ====== traversal_flush timer task ====== [2024-09-26 16:00:25.619232] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d76030, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6163825430799493735, dag_cnt=1, dag_type_cnts=1) [2024-09-26 16:00:25.619240] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] success to create and add dag(ret=0, dag=0x7f9264d76030) [2024-09-26 16:00:25.619243] INFO [STORAGE] flush (ob_tx_data_memtable.cpp:917) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] [TX DATA MERGE]schedule flush tx data memtable task done(ret=0, ret="OB_SUCCESS", get_ls_id()={id:1}, this=0x7f92a2294080, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:4, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337625619243, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:25.619271] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] succeed to add sys task(task={start_time:1727337625619258, task_id:YB42C0A8002D-0006230056D8C5BA-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.619280] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=7] schedule one task(task=0x7f9264d763f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.619289] INFO [MDS] flush (mds_table_mgr.cpp:172) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=29] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:25.619311] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.619316] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.619318] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726157244792838000) [2024-09-26 16:00:25.619321] INFO [STORAGE] ready_for_flush (ob_tx_data_memtable.cpp:865) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=2] memtable is frozen yet.(this=0x7f92a2294080) [2024-09-26 16:00:25.619328] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=22] add dag success(dag=0x7f9264d76560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6659188001345271846, dag_cnt=2, dag_type_cnts=1) [2024-09-26 16:00:25.619332] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d76560) [2024-09-26 16:00:25.619335] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, create_flush_dag_time:1727337625619335, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452867, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200240}, checkpoint_unit_ptr:0x7f92a1a70850}, is_freeze_clock:true}) [2024-09-26 16:00:25.619342] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337625619319, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619359] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=14] add dag success(dag=0x7f9264d76a90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-9069613569758001279, dag_cnt=3, dag_type_cnts=2) [2024-09-26 16:00:25.619361] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d76a90) [2024-09-26 16:00:25.619362] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, create_flush_dag_time:1727337625619362, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452892, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200489}, checkpoint_unit_ptr:0x7f92a1a70080}, is_freeze_clock:true}) [2024-09-26 16:00:25.619366] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337625619356, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619375] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d76fc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1288986994123435108, dag_cnt=4, dag_type_cnts=3) [2024-09-26 16:00:25.619377] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d76fc0) [2024-09-26 16:00:25.619379] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337625619378, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452911, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847222}, checkpoint_unit_ptr:0x7f92a1a71020}, is_freeze_clock:true}) [2024-09-26 16:00:25.619382] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337625619373, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619390] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d774f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5355160120942094702, dag_cnt=5, dag_type_cnts=4) [2024-09-26 16:00:25.619392] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d774f0) [2024-09-26 16:00:25.619393] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337625619393, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452929, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847223}, checkpoint_unit_ptr:0x7f92a1a717f0}, is_freeze_clock:true}) [2024-09-26 16:00:25.619397] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337625619388, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619419] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337625619416, task_id:YB42C0A8002D-0006230056D8C5BB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.619424] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=4] schedule one task(task=0x7f9264d76920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.619430] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625619429, task_id:YB42C0A8002D-0006230056D8C5BC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.619433] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=2] schedule one task(task=0x7f9264d76e50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.619436] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337625619435, task_id:YB42C0A8002D-0006230056D8C5BD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.619438] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=2] schedule one task(task=0x7f9264d77380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.619443] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625619442, task_id:YB42C0A8002D-0006230056D8C5BE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.619445] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=2] schedule one task(task=0x7f9264d778b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.619478] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.619481] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.619483] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:25.619486] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.619488] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.619490] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.619492] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:25.619493] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:25.619497] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.619504] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.619524] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.619526] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.619528] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:25.619530] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.619533] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847222}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:25.619541] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.619556] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.619561] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.619563] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:25.619566] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.619582] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405042) [2024-09-26 16:00:25.619595] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:25.619603] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405070) [2024-09-26 16:00:25.619610] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:25.619614] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405087) [2024-09-26 16:00:25.619620] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:25.619623] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405104) [2024-09-26 16:00:25.619629] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:25.619635] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] add dag success(dag=0x7f9264d77a20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1994541446894784994, dag_cnt=6, dag_type_cnts=5) [2024-09-26 16:00:25.619638] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d77a20) [2024-09-26 16:00:25.619640] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, create_flush_dag_time:1727337625619640, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209556884, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:226}, checkpoint_unit_ptr:0x7f92a1a73f00}, is_freeze_clock:true}) [2024-09-26 16:00:25.619646] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337625619630, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619662] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625619660, task_id:YB42C0A8002D-0006230056D8C5BF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.619667] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=4] schedule one task(task=0x7f9264d77de0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.619676] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=20] add dag success(dag=0x7f9264d77f50, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8791616776033068452, dag_cnt=7, dag_type_cnts=6) [2024-09-26 16:00:25.619679] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d77f50) [2024-09-26 16:00:25.619681] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337625619680, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405141, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847408}, checkpoint_unit_ptr:0x7f92a1a746d0}, is_freeze_clock:true}) [2024-09-26 16:00:25.619686] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.619689] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.619684] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337625619666, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619691] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.619695] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d78480, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6550230011160095889, dag_cnt=8, dag_type_cnts=7) [2024-09-26 16:00:25.619695] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.619697] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d78480) [2024-09-26 16:00:25.619698] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:00:25.619698] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337625619698, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405158, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847409}, checkpoint_unit_ptr:0x7f92a1a75670}, is_freeze_clock:true}) [2024-09-26 16:00:25.619704] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241777443612000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.619702] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337625619691, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619711] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d789b0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6809498982591245614, dag_cnt=9, dag_type_cnts=8) [2024-09-26 16:00:25.619712] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d789b0) [2024-09-26 16:00:25.619714] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, create_flush_dag_time:1727337625619714, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405174, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:200004}, checkpoint_unit_ptr:0x7f92a1a74ea0}, is_freeze_clock:true}) [2024-09-26 16:00:25.619717] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337625619708, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619728] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d78ee0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6815944285528637599, dag_cnt=10, dag_type_cnts=9) [2024-09-26 16:00:25.619730] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d78ee0) [2024-09-26 16:00:25.619732] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, create_flush_dag_time:1727337625619731, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209506214, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1}, checkpoint_unit_ptr:0x7f92a1a76610}, is_freeze_clock:true}) [2024-09-26 16:00:25.619734] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337625619723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619743] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d79410, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-4182959058374342994, dag_cnt=11, dag_type_cnts=10) [2024-09-26 16:00:25.619753] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] success to create and add dag(ret=0, dag=0x7f9264d79410) [2024-09-26 16:00:25.619755] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, create_flush_dag_time:1727337625619755, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405204, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:412}, checkpoint_unit_ptr:0x7f92a1a75e40}, is_freeze_clock:true}) [2024-09-26 16:00:25.619758] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337625619741, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619770] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d79940, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1586518267935340, dag_cnt=12, dag_type_cnts=11) [2024-09-26 16:00:25.619775] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] success to create and add dag(ret=0, dag=0x7f9264d79940) [2024-09-26 16:00:25.619784] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, create_flush_dag_time:1727337625619784, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405220, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:444}, checkpoint_unit_ptr:0x7f92a1a775b0}, is_freeze_clock:true}) [2024-09-26 16:00:25.619788] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337625619765, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619801] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d79e70, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3044367941103776476, dag_cnt=13, dag_type_cnts=12) [2024-09-26 16:00:25.619803] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d79e70) [2024-09-26 16:00:25.619805] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, create_flush_dag_time:1727337625619805, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455964, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:309}, checkpoint_unit_ptr:0x7f92a1a76de0}, is_freeze_clock:true}) [2024-09-26 16:00:25.619808] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337625619795, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619819] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7a3a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1596830851104282047, dag_cnt=14, dag_type_cnts=13) [2024-09-26 16:00:25.619823] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7a3a0) [2024-09-26 16:00:25.619825] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, create_flush_dag_time:1727337625619825, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455986, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:331}, checkpoint_unit_ptr:0x7f92a1a77d80}, is_freeze_clock:true}) [2024-09-26 16:00:25.619828] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337625619815, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619840] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d7a8d0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2360195962081676673, dag_cnt=15, dag_type_cnts=14) [2024-09-26 16:00:25.619842] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7a8d0) [2024-09-26 16:00:25.619844] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, create_flush_dag_time:1727337625619844, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405262, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:458}, checkpoint_unit_ptr:0x7f92a1a7a490}, is_freeze_clock:true}) [2024-09-26 16:00:25.619847] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337625619837, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619859] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7ae00, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5675615614797020460, dag_cnt=16, dag_type_cnts=15) [2024-09-26 16:00:25.619861] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7ae00) [2024-09-26 16:00:25.619863] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, create_flush_dag_time:1727337625619863, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405279, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:487}, checkpoint_unit_ptr:0x7f92a1a79cc0}, is_freeze_clock:true}) [2024-09-26 16:00:25.619866] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337625619854, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619876] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7b330, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-612435342908088006, dag_cnt=17, dag_type_cnts=16) [2024-09-26 16:00:25.619880] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7b330) [2024-09-26 16:00:25.619882] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, create_flush_dag_time:1727337625619882, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405296, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:455}, checkpoint_unit_ptr:0x7f92a1a794f0}, is_freeze_clock:true}) [2024-09-26 16:00:25.619885] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337625619873, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619896] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d7b860, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7868818205990873505, dag_cnt=18, dag_type_cnts=17) [2024-09-26 16:00:25.619898] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7b860) [2024-09-26 16:00:25.619900] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, create_flush_dag_time:1727337625619900, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405313, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:492}, checkpoint_unit_ptr:0x7f92a1a78550}, is_freeze_clock:true}) [2024-09-26 16:00:25.619903] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337625619893, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619914] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7bd90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7071731214611448811, dag_cnt=19, dag_type_cnts=18) [2024-09-26 16:00:25.619920] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] success to create and add dag(ret=0, dag=0x7f9264d7bd90) [2024-09-26 16:00:25.619922] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, create_flush_dag_time:1727337625619922, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405347, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:501}, checkpoint_unit_ptr:0x7f92a1a78d20}, is_freeze_clock:true}) [2024-09-26 16:00:25.619925] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337625619910, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619936] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7c2c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8731750006000903454, dag_cnt=20, dag_type_cnts=19) [2024-09-26 16:00:25.619939] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d7c2c0) [2024-09-26 16:00:25.619941] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, create_flush_dag_time:1727337625619941, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405365, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:472}, checkpoint_unit_ptr:0x7f92a1a7ac60}, is_freeze_clock:true}) [2024-09-26 16:00:25.619944] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337625619932, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619953] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7c7f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6766388540859538791, dag_cnt=21, dag_type_cnts=20) [2024-09-26 16:00:25.619957] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7c7f0) [2024-09-26 16:00:25.619959] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, create_flush_dag_time:1727337625619959, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405384, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:342}, checkpoint_unit_ptr:0x7f92a1a7cba0}, is_freeze_clock:true}) [2024-09-26 16:00:25.619962] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337625619950, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619974] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264d7cd20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2986199386776543176, dag_cnt=22, dag_type_cnts=21) [2024-09-26 16:00:25.619976] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7cd20) [2024-09-26 16:00:25.619977] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337625619977, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405418, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:140}, checkpoint_unit_ptr:0x7f92a1a7b430}, is_freeze_clock:true}) [2024-09-26 16:00:25.619981] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337625619971, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.619991] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7d250, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6947063304478012270, dag_cnt=23, dag_type_cnts=22) [2024-09-26 16:00:25.619996] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] success to create and add dag(ret=0, dag=0x7f9264d7d250) [2024-09-26 16:00:25.619998] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337625619998, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405436, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101009}, checkpoint_unit_ptr:0x7f92a1a7bc00}, is_freeze_clock:true}) [2024-09-26 16:00:25.620001] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337625619987, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620011] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7d780, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5843378789243662202, dag_cnt=24, dag_type_cnts=23) [2024-09-26 16:00:25.620015] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7d780) [2024-09-26 16:00:25.620017] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337625620017, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405453, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101010}, checkpoint_unit_ptr:0x7f92a1a7c3d0}, is_freeze_clock:true}) [2024-09-26 16:00:25.620020] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337625620008, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620032] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d7dcb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1841823684513746176, dag_cnt=25, dag_type_cnts=24) [2024-09-26 16:00:25.620034] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7dcb0) [2024-09-26 16:00:25.620036] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, create_flush_dag_time:1727337625620036, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405470, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:323}, checkpoint_unit_ptr:0x7f92a1a7d370}, is_freeze_clock:true}) [2024-09-26 16:00:25.620039] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337625620028, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620050] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7e1e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-2678407261265885087, dag_cnt=26, dag_type_cnts=25) [2024-09-26 16:00:25.620052] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7e1e0) [2024-09-26 16:00:25.620054] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, create_flush_dag_time:1727337625620054, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405487, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:377}, checkpoint_unit_ptr:0x7f92a1a7e310}, is_freeze_clock:true}) [2024-09-26 16:00:25.620057] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337625620046, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620068] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7e710, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7175089551615529855, dag_cnt=27, dag_type_cnts=26) [2024-09-26 16:00:25.620073] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] success to create and add dag(ret=0, dag=0x7f9264d7e710) [2024-09-26 16:00:25.620075] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, create_flush_dag_time:1727337625620075, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405503, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:297}, checkpoint_unit_ptr:0x7f92a1a7eae0}, is_freeze_clock:true}) [2024-09-26 16:00:25.620078] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337625620063, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620088] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7ec40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3730526293242960203, dag_cnt=28, dag_type_cnts=27) [2024-09-26 16:00:25.620092] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7ec40) [2024-09-26 16:00:25.620094] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337625620094, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405519, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101065}, checkpoint_unit_ptr:0x7f92a1a7db40}, is_freeze_clock:true}) [2024-09-26 16:00:25.620097] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337625620085, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620108] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d7f170, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=4663792628280492528, dag_cnt=29, dag_type_cnts=28) [2024-09-26 16:00:25.620110] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7f170) [2024-09-26 16:00:25.620112] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337625620112, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405536, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:333}, checkpoint_unit_ptr:0x7f92a3bc4850}, is_freeze_clock:true}) [2024-09-26 16:00:25.620125] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=13] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337625620105, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620139] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d7f6a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8399557287176188414, dag_cnt=30, dag_type_cnts=29) [2024-09-26 16:00:25.620142] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d7f6a0) [2024-09-26 16:00:25.620144] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337625620143, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405552, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101066}, checkpoint_unit_ptr:0x7f92a3bc5020}, is_freeze_clock:true}) [2024-09-26 16:00:25.620147] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337625620133, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620157] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7fbd0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8068938909778324349, dag_cnt=31, dag_type_cnts=30) [2024-09-26 16:00:25.620161] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7fbd0) [2024-09-26 16:00:25.620163] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, create_flush_dag_time:1727337625620163, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405569, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:334}, checkpoint_unit_ptr:0x7f92a3bc57f0}, is_freeze_clock:true}) [2024-09-26 16:00:25.620166] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337625620154, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620179] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d80100, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7665372330263152188, dag_cnt=32, dag_type_cnts=31) [2024-09-26 16:00:25.620181] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d80100) [2024-09-26 16:00:25.620182] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, create_flush_dag_time:1727337625620182, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405586, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:451}, checkpoint_unit_ptr:0x7f92a3bc4080}, is_freeze_clock:true}) [2024-09-26 16:00:25.620185] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337625620174, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620196] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d80630, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6759548225246904317, dag_cnt=33, dag_type_cnts=32) [2024-09-26 16:00:25.620199] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d80630) [2024-09-26 16:00:25.620200] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, create_flush_dag_time:1727337625620200, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405602, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:378}, checkpoint_unit_ptr:0x7f92a1a7f2b0}, is_freeze_clock:true}) [2024-09-26 16:00:25.620203] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337625620192, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620216] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d80b60, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8944614441376604790, dag_cnt=34, dag_type_cnts=33) [2024-09-26 16:00:25.620221] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] success to create and add dag(ret=0, dag=0x7f9264d80b60) [2024-09-26 16:00:25.620222] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, create_flush_dag_time:1727337625620222, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209456003, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:121}, checkpoint_unit_ptr:0x7f92a3bc5fc0}, is_freeze_clock:true}) [2024-09-26 16:00:25.620225] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337625620210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620235] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d81090, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=507863347806783454, dag_cnt=35, dag_type_cnts=34) [2024-09-26 16:00:25.620240] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d81090) [2024-09-26 16:00:25.620241] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, create_flush_dag_time:1727337625620241, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405633, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:215}, checkpoint_unit_ptr:0x7f92a3bc86d0}, is_freeze_clock:true}) [2024-09-26 16:00:25.620244] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337625620232, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620256] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264d815c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1604282355669434429, dag_cnt=36, dag_type_cnts=35) [2024-09-26 16:00:25.620258] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d815c0) [2024-09-26 16:00:25.620259] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, create_flush_dag_time:1727337625620259, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405649, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:456}, checkpoint_unit_ptr:0x7f92a3bc6790}, is_freeze_clock:true}) [2024-09-26 16:00:25.620262] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337625620253, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620274] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d81af0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3553649416163308074, dag_cnt=37, dag_type_cnts=36) [2024-09-26 16:00:25.620276] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d81af0) [2024-09-26 16:00:25.620278] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337625620278, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405666, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:154}, checkpoint_unit_ptr:0x7f92a3bc6f60}, is_freeze_clock:true}) [2024-09-26 16:00:25.620281] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337625620269, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620291] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d82020, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8097406237500329300, dag_cnt=38, dag_type_cnts=37) [2024-09-26 16:00:25.620295] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d82020) [2024-09-26 16:00:25.620296] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337625620296, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405682, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101015}, checkpoint_unit_ptr:0x7f92a3bc7730}, is_freeze_clock:true}) [2024-09-26 16:00:25.620299] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337625620288, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620311] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264d82550, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3976350754940852513, dag_cnt=39, dag_type_cnts=38) [2024-09-26 16:00:25.620313] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d82550) [2024-09-26 16:00:25.620315] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337625620315, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405699, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101016}, checkpoint_unit_ptr:0x7f92a3bc7f00}, is_freeze_clock:true}) [2024-09-26 16:00:25.620318] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337625620308, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620330] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d82a80, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8720304653345806796, dag_cnt=40, dag_type_cnts=39) [2024-09-26 16:00:25.620332] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d82a80) [2024-09-26 16:00:25.620334] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, create_flush_dag_time:1727337625620334, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405716, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101063}, checkpoint_unit_ptr:0x7f92a3bca610}, is_freeze_clock:true}) [2024-09-26 16:00:25.620337] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337625620325, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620348] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d82fb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1830562711513853292, dag_cnt=41, dag_type_cnts=40) [2024-09-26 16:00:25.620351] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d82fb0) [2024-09-26 16:00:25.620352] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, create_flush_dag_time:1727337625620352, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405733, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:452}, checkpoint_unit_ptr:0x7f92a3bc9e40}, is_freeze_clock:true}) [2024-09-26 16:00:25.620355] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337625620344, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620365] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d834e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1536604874905918360, dag_cnt=42, dag_type_cnts=41) [2024-09-26 16:00:25.620369] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d834e0) [2024-09-26 16:00:25.620371] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337625620371, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405750, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:328}, checkpoint_unit_ptr:0x7f92a3bc8ea0}, is_freeze_clock:true}) [2024-09-26 16:00:25.620374] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337625620362, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620386] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d83a10, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6933628521298738662, dag_cnt=43, dag_type_cnts=42) [2024-09-26 16:00:25.620388] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d83a10) [2024-09-26 16:00:25.620389] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337625620389, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405767, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:330}, checkpoint_unit_ptr:0x7f92a3bc9670}, is_freeze_clock:true}) [2024-09-26 16:00:25.620392] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337625620383, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620403] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d83f40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1056199536334423564, dag_cnt=44, dag_type_cnts=43) [2024-09-26 16:00:25.620422] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d83f40) [2024-09-26 16:00:25.620425] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337625620425, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405784, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:329}, checkpoint_unit_ptr:0x7f92a3bcb5b0}, is_freeze_clock:true}) [2024-09-26 16:00:25.620428] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337625620399, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620439] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d84470, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7405463607918801432, dag_cnt=45, dag_type_cnts=44) [2024-09-26 16:00:25.620444] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d84470) [2024-09-26 16:00:25.620446] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, create_flush_dag_time:1727337625620446, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405801, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:332}, checkpoint_unit_ptr:0x7f92a3bcade0}, is_freeze_clock:true}) [2024-09-26 16:00:25.620449] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337625620435, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620472] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=19] add dag success(dag=0x7f9264d849a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6699723950958743903, dag_cnt=46, dag_type_cnts=45) [2024-09-26 16:00:25.620477] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d849a0) [2024-09-26 16:00:25.620479] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, create_flush_dag_time:1727337625620479, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405818, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101067}, checkpoint_unit_ptr:0x7f92a3bcbd80}, is_freeze_clock:true}) [2024-09-26 16:00:25.620482] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337625620468, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.620490] INFO [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:655) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] traversal_flush successfully(ls_->get_ls_id()={id:1}, flush_tasks=[{table:0x7f92a1a70850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a70080, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71020, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a717f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71fc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72790, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72f60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73730, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73f00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a746d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75670, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a74ea0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76610, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75e40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a775b0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76de0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a77d80, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7a490, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a79cc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a794f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78550, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78d20, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7ac60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7cba0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7b430, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7bc00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7c3d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7d370, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7e310, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7eae0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7db40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a3bc4850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}]) [2024-09-26 16:00:25.620549] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=57] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-09-26 16:00:25.627031] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:25.627034] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:25.627046] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337625627028) [2024-09-26 16:00:25.627070] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337625526486, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:25.627084] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223354862565, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.627090] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:25.627097] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:25.627102] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:25.627105] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:25.627122] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.627125] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.627127] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337625627074) [2024-09-26 16:00:25.627139] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337625288788]}) [2024-09-26 16:00:25.630069] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1, storage_schema:{this:0x7f92728f12b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.630086] INFO [STORAGE] prepare_tx_data_list (ob_tx_data_memtable.cpp:266) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=16] construct list more than once. skip this time(this=0x7f92a2294080) [2024-09-26 16:00:25.630879] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:25.630885] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.634824] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.634861] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=37] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337625619373, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:25.634871] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.634900] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337625619630, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.634917] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752755184, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828752755184, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.634945] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=12] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.634952] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.634972] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=13] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:25.634962] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:226}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752755184, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.634976] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.634983] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847222}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:25.634988] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:226}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752755184, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:226}, table_count:1, [{i:0, table_key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}, merge_scn:{val:1726241777443612000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625619683, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=15.27ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.635003] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:1152921504606847222}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847222}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625619522, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=15.46ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.635024] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=26] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BF-0-0, dag_ret:0, dag_status:2, start_time:1727337625619666, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}, scanned_row_cnt_arr:0x7f9287db7480, output_block_cnt_arr:0x7f9287db7488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:56623104, latest_update_ts:1727337625619683, estimated_finish_time:1727337642635051}) [2024-09-26 16:00:25.635033] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264d77de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BF-0-0, dag_ret:0, dag_status:2, start_time:1727337625619666, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:25.635041] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=8] task finish process(ret=0, start_time=1727337625619681, end_time=1727337625635041, runtime=15360, *this={this:0x7f9264d77de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BF-0-0, dag_ret:0, dag_status:2, start_time:1727337625619666, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:25.635212] INFO [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:329) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=3] pre-process commit versions row(get_ls_id()={id:1}, do_recycle_=true, current_time=1727337625635210, prev_recycle_time=0) [2024-09-26 16:00:25.635268] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=21] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:25.635998] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=4] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=8366, total_rows=58562) [2024-09-26 16:00:25.636448] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=9] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BD-0-0, dag_ret:0, dag_status:2, start_time:1727337625619438, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b23470, output_block_cnt_arr:0x7f9264b23478, concurrent_cnt:1, estimate_row_cnt:329211, estimate_occupy_size:150994944, latest_update_ts:1727337625619522, estimated_finish_time:1727337642636519}) [2024-09-26 16:00:25.636482] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=33] succeed to init merge ctx(task={this:0x7f9264d77380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BD-0-0, dag_ret:0, dag_status:2, start_time:1727337625619438, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.636489] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=7] task finish process(ret=0, start_time=1727337625619521, end_time=1727337625636489, runtime=16968, *this={this:0x7f9264d77380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BD-0-0, dag_ret:0, dag_status:2, start_time:1727337625619438, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.636534] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726158220122116000) [2024-09-26 16:00:25.636551] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=7] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.636585] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:25.636591] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.637462] INFO [STORAGE] get_ls_min_end_scn (ob_ls_tablet_service.cpp:5618) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=9] get ls min end scn finish(ls_checkpoint={val:1726163586617842000, v:0}) [2024-09-26 16:00:25.637479] WDIAG [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:761) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=11][errcode=-4023] failed to get snapshot version(ret=-4023, MTL_ID()=1) [2024-09-26 16:00:25.637485] INFO [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:773) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=5] get tx data recycle scn finish.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, recycle_scn={val:1726076389100953000, v:0}, min_end_scn_from_old_tablets={val:4611686018427387903, v:0}, min_end_scn_from_latest_tablets={val:1726076389100953000, v:0}) [2024-09-26 16:00:25.637490] WDIAG [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:336) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=4][errcode=-4023] get recycle ts failed.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:25.637495] WDIAG [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:236) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=2][errcode=-4023] process commit version row failed.(ret=-4023, ret="OB_EAGAIN", this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337625619243, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:25.637508] INFO [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:254) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=12] [TX DATA MERGE]pre process for merge done(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}, this=0x7f92a2294080, tg=time guard have no click events for optimization, fake_tx_data_guard={tx_data:{tx_id:{txid:9223372036854775807}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337625619243, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:25.637529] WDIAG [STORAGE.COMPACTION] pre_process_tx_data_table_merge_ (ob_tx_table_merge_task.cpp:117) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=20][errcode=-4023] do pre process for tx data table merge failed.(ret=-4023, ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, table={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337625619243, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:25.637540] WDIAG [STORAGE.COMPACTION] inner_init_ctx (ob_tx_table_merge_task.cpp:157) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=10][errcode=-4023] pre process tx data table for merge failed.(ret=-4023, ret="OB_EAGAIN", ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.637544] WDIAG [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1061) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=3][errcode=-4023] fail to inner init ctx(ret=-4023, tablet_id={id:49402}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, create_snapshot_version:0, is_full_merge:false, merge_level:1, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:4, range_array:[], concurrent_cnt:0, is_inited:false}, schema_ctx:{schema_version:1, storage_schema:{this:0x7f92728f12b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:49402}, table_count:1, [{i:0, table_key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref:4}]}, schedule_major:false, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}, merge_scn:{val:4611686018427387903, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:NULL, compaction_filter:NULL, time_guard:total=18.23ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}) [2024-09-26 16:00:25.637587] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:975) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=43][errcode=-4023] failed to build merge ctx(ret=-4023, ctx->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.637591] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:994) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=4][errcode=-4023] sstable merge finish(ret=-4023, ctx=0x7f92728f0060, task={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C5BA-0-0, dag_ret:0, dag_status:2, start_time:1727337625619279, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:25.637598] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=7][errcode=-4023] failed to process task(ret=-4023) [2024-09-26 16:00:25.637601] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=2] task finish process(ret=-4023, start_time=1727337625619302, end_time=1727337625637600, runtime=18298, *this={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C5BA-0-0, dag_ret:0, dag_status:2, start_time:1727337625619279, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:25.637608] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=6][errcode=-4023] failed to do work(ret=-4023, *task_={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C5BA-0-0, dag_ret:0, dag_status:2, start_time:1727337625619279, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:25.637616] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=5] dag finished(dag_ret=-4023, runtime=18337, dag_cnt=45, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264d76030, dag={ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C5BA-0-0, dag_ret:-4023, dag_status:5, start_time:1727337625619279, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}) [2024-09-26 16:00:25.637627] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5BA-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337625619258, task_id:YB42C0A8002D-0006230056D8C5BA-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.637707] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.637722] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=15] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337625619388, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:25.637730] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.637831] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=7] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:25.637836] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.637845] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847223}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:25.637868] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:1152921504606847223}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847223}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625619472, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=18.37ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.638104] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=20] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:25.638791] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=7999, total_rows=55993) [2024-09-26 16:00:25.639164] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=6] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BE-0-0, dag_ret:0, dag_status:2, start_time:1727337625619445, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265adb470, output_block_cnt_arr:0x7f9265adb478, concurrent_cnt:1, estimate_row_cnt:332829, estimate_occupy_size:150994944, latest_update_ts:1727337625619472, estimated_finish_time:1727337642639236}) [2024-09-26 16:00:25.639180] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=15] succeed to init merge ctx(task={this:0x7f9264d778b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BE-0-0, dag_ret:0, dag_status:2, start_time:1727337625619445, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.639187] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=6] task finish process(ret=0, start_time=1727337625619470, end_time=1727337625639187, runtime=19717, *this={this:0x7f9264d778b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BE-0-0, dag_ret:0, dag_status:2, start_time:1727337625619445, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.639241] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=2] schedule one task(task=0x7f9264af8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.639253] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=8] schedule one task(task=0x7f9264ae83c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.639257] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=2] schedule one task(task=0x7f9264ae8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.639263] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625639260, task_id:YB42C0A8002D-0006230056D8C5C0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.639268] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=5] schedule one task(task=0x7f9264d78310, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.639292] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.639296] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.639299] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.639305] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.639391] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b58060, count:4, allocator:0x7f92a339c7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263b58350, local_datums:0x7f9263b58350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:25.639414] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=20] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:4, allocator:0x7f92a339d558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:25.639427] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=226, range=0x7f92aa352200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b00060, count:5, allocator:0x7f92c195e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9265b00360, local_datums:0x7f9265b00360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:25.639684] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=7][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:25.639746] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.639751] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.639759] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:25.639762] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:25.639768] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=5][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9263a3e140, hex: 3230323430393131303133333131, cstr: 20240911013311},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9263a3e14e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e0e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-15038, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7227, data_checksum:1452123993, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265892030, buf_size:7227, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1818264584, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:00:25.639784] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=15][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:25.639786] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:25.639787] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:25.639790] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=3][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:25.639792] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263b58350, local_datums:0x7f9263b58350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b583d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b58408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263b58350, local_datums:0x7f9263b58350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b583d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b58408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92a339c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a339c118, limit_param:null, stmt_allocator:0x7f92a339c7d8, allocator:0x7f92a339c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320384, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263b58350, local_datums:0x7f9263b58350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b583d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b58408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:00:25.639895] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=103][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:25.639898] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=3][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263b58350, local_datums:0x7f9263b58350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b583d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b58408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92a339c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a339c118, limit_param:null, stmt_allocator:0x7f92a339c7d8, allocator:0x7f92a339c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320384, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263b58350, local_datums:0x7f9263b58350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b583d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b58408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:00:25.639932] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=34][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:25.639935] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=1] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.639949] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=9] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337625619388, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.639959] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:25.639974] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=3][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:25.639978] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:25.639979] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337625639323, end_time=1727337625639979, runtime=656, *this={this:0x7f9264ae83c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BE-0-0, dag_ret:0, dag_status:2, start_time:1727337625619445, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.639986] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264ae83c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BE-0-0, dag_ret:0, dag_status:2, start_time:1727337625619445, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:25.639995] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=20550, dag_cnt=44, dag_cnts_[dag.get_type()]=44, &dag=0x7f9264d774f0, dag={ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BE-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625619445, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.640005] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BE-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337625619442, task_id:YB42C0A8002D-0006230056D8C5BE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.640055] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625640053, task_id:YB42C0A8002D-0006230056D8C5C1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.640061] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=5] schedule one task(task=0x7f9264d78840, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.640085] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.640089] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.640091] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.640096] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.643691] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3c060, count:4, allocator:0x7f92691a27d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3c350, local_datums:0x7f9263a3c350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:25.643714] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=19] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:4, allocator:0x7f92691a3558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263b5e350, local_datums:0x7f9263b5e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:25.643987] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=7][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:25.644042] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=8][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.644046] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.644054] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:25.644057] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:25.644062] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=4][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9264d62140, hex: 3230323430393131303133333438, cstr: 20240911013348},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9264d6214e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d620e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3080, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7221, data_checksum:4284205003, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9263bde030, buf_size:7221, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1004563202, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:00:25.644076] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=13][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:25.644080] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=4][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:25.644082] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:25.644084] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:25.644086] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3c350, local_datums:0x7f9263a3c350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3c350, local_datums:0x7f9263a3c350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92691a2118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691a2118, limit_param:null, stmt_allocator:0x7f92691a27d8, allocator:0x7f92691a27d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3c350, local_datums:0x7f9263a3c350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:00:25.644142] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=56][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:25.644144] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3c350, local_datums:0x7f9263a3c350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92691a2118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691a2118, limit_param:null, stmt_allocator:0x7f92691a27d8, allocator:0x7f92691a27d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3c350, local_datums:0x7f9263a3c350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:00:25.644176] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=32][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:25.644178] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=1] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.644192] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=9] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337625619373, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.644201] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:25.644212] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:25.644216] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:25.644218] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337625643640, end_time=1727337625644218, runtime=578, *this={this:0x7f9264af8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BD-0-0, dag_ret:0, dag_status:2, start_time:1727337625619438, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.644226] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264af8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BD-0-0, dag_ret:0, dag_status:2, start_time:1727337625619438, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:25.644235] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=24796, dag_cnt=43, dag_cnts_[dag.get_type()]=43, &dag=0x7f9264d76fc0, dag={ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BD-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625619438, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.644243] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BD-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625619435, task_id:YB42C0A8002D-0006230056D8C5BD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.644366] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.644376] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=10] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337625619356, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=17, max_schema_version_on_memtable=1725120002010696) [2024-09-26 16:00:25.644384] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=17, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.644500] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1887, total_rows=92463) [2024-09-26 16:00:25.644505] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.644512] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200489}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:19, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.644543] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:200489}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200489}, table_count:2, [{i:0, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625619481, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=25.03ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.644832] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=23] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BC-0-0, dag_ret:0, dag_status:2, start_time:1727337625619432, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f460, output_block_cnt_arr:0x7f9264b1f468, concurrent_cnt:1, estimate_row_cnt:254187, estimate_occupy_size:138412032, latest_update_ts:1727337625619481, estimated_finish_time:1727337642644898}) [2024-09-26 16:00:25.644846] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=14] succeed to init merge ctx(task={this:0x7f9264d76e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BC-0-0, dag_ret:0, dag_status:2, start_time:1727337625619432, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.644852] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=6] task finish process(ret=0, start_time=1727337625619479, end_time=1727337625644852, runtime=25373, *this={this:0x7f9264d76e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BC-0-0, dag_ret:0, dag_status:2, start_time:1727337625619432, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.644891] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=3] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.644901] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337625644899, task_id:YB42C0A8002D-0006230056D8C5C2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.644906] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=4] schedule one task(task=0x7f9264d78d70, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.644925] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.644929] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.644932] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.644937] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.644941] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.644955] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.645025] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92ab056200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:3, allocator:0x7f92636767d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:25.645043] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=15] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92ab056200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fc060, count:3, allocator:0x7f9263677558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fc320, local_datums:0x7f92665fc320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:25.645208] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:25.645254] ERROR issue_dba_error (ob_log.cpp:1875) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.645258] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.645266] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:25.645269] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:25.645271] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76070, hex: 9F094D0300000000, int: 55380383},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c760a8, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c760e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3044, column_count:19, rowkey_column_count:3, has_column_checksum:0, row_count:83, row_store_type:0, opt:5, var_column_count:0, row_offset:16109, original_length:16381, max_merged_trans_version:1726163995623997000, data_length:16381, data_zlength:7759, data_checksum:3416849541, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92684de030, buf_size:7759, data_size:16381, row_count:83, column_count:19, max_merged_trans_version:1726163995623997000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3537734793, row_count_delta:83, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16381}) [2024-09-26 16:00:25.645282] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:25.645284] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:25.645287] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=3][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:25.645289] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:25.645291] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fc320, local_datums:0x7f92665fc320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fc320, local_datums:0x7f92665fc320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f9263676e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263676e98, limit_param:null, stmt_allocator:0x7f9263677558, allocator:0x7f9263677558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fc320, local_datums:0x7f92665fc320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337625619356, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}) [2024-09-26 16:00:25.645359] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=67][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:25.645361] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fc320, local_datums:0x7f92665fc320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f9263676e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263676e98, limit_param:null, stmt_allocator:0x7f9263677558, allocator:0x7f9263677558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fc320, local_datums:0x7f92665fc320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337625619356, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}]) [2024-09-26 16:00:25.645429] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=68][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:25.645432] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=2] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.645444] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=9] partition merge iter row count(i=1, row_count=83, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, table={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337625619356, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.645454] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:25.645465] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:25.645469] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:25.645470] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337625644978, end_time=1727337625645470, runtime=492, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BC-0-0, dag_ret:0, dag_status:2, start_time:1727337625619432, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.645478] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BC-0-0, dag_ret:0, dag_status:2, start_time:1727337625619432, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:25.645485] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=26053, dag_cnt=42, dag_cnts_[dag.get_type()]=42, &dag=0x7f9264d76a90, dag={ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BC-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625619432, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.645493] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BC-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337625619429, task_id:YB42C0A8002D-0006230056D8C5BC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.645634] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625645632, task_id:YB42C0A8002D-0006230056D8C5C3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.645641] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=6] schedule one task(task=0x7f9264d792a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.645712] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.645719] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.645722] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.645727] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.645731] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.645741] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241621541056000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.646095] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=19][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:25.646154] ERROR issue_dba_error (ob_log.cpp:1875) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.646159] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.646167] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:25.646170] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:25.646172] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c0a8, hex: 0000000000000000, int: 0},idx=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9265b0c0e0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c118, hex: 58A3C56A4A280BE8, int: -1726241732082425000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-19590, column_count:10, rowkey_column_count:5, has_column_checksum:0, row_count:224, row_store_type:0, opt:0, var_column_count:0, row_offset:15527, original_length:16363, max_merged_trans_version:1726241777443612000, data_length:16363, data_zlength:16363, data_checksum:2057046089, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926fc04090, buf_size:16363, data_size:16363, row_count:224, column_count:10, max_merged_trans_version:1726241777443612000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1006468742, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16363}) [2024-09-26 16:00:25.646185] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:25.646187] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:25.646188] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:25.646193] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=5][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:25.646195] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:25.646197] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:25.646198] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9265b00360, local_datums:0x7f9265b00360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b003a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9265b003e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9265b00360, local_datums:0x7f9265b00360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b003a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9265b003e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f92c195e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c195e118, limit_param:null, stmt_allocator:0x7f92c195e7d8, allocator:0x7f92c195e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:327190772997047620, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9265b00360, local_datums:0x7f9265b00360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b003a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9265b003e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337625619630, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}) [2024-09-26 16:00:25.646259] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=61][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:25.646261] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9265b00360, local_datums:0x7f9265b00360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b003a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9265b003e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f92c195e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c195e118, limit_param:null, stmt_allocator:0x7f92c195e7d8, allocator:0x7f92c195e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:327190772997047620, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9265b00360, local_datums:0x7f9265b00360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b003a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9265b003e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b00450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337625619630, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}]) [2024-09-26 16:00:25.646296] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=35][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:25.646299] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=2] partition merge iter row count(i=0, row_count=224, ghost_row_count=0, pkey={tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, table={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337625619630, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.646312] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:25.646323] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:25.646327] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:25.646328] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337625639371, end_time=1727337625646328, runtime=6957, *this={this:0x7f9264ae8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BF-0-0, dag_ret:0, dag_status:2, start_time:1727337625619666, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:25.646336] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264ae8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BF-0-0, dag_ret:0, dag_status:2, start_time:1727337625619666, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:25.646344] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=26677, dag_cnt=41, dag_cnts_[dag.get_type()]=41, &dag=0x7f9264d77a20, dag={ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BF-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625619666, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}) [2024-09-26 16:00:25.646352] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BF-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625619660, task_id:YB42C0A8002D-0006230056D8C5BF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.646379] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337625646377, task_id:YB42C0A8002D-0006230056D8C5C4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.646383] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=4] schedule one task(task=0x7f9264d797d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.646407] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.646431] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=24][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.646434] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.646439] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.646450] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.646460] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=10] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337625619319, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=18, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:25.646479] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=19] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.646443] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726176799210367000, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b862fe60, is_inited:true, wash_score:1727337625618278000, hold_ref_cnt:true, ref_cnt:2, version:2, length:497, tablet_addr:[816017-1849-0-2305843637353291776-2](offset=630784,size=585,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:412}, data_tablet_id:{id:412}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726176799210367000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3559841642, row_count:4}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8630518, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844255828574208-0](offset=622592,size=1161,type=2,seq=0), type:"MAJOR", tablet_id:{id:412}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725843390668168000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335631832201-0](offset=623753,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:412}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337625619741, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd6790, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334211402085}, this:0x7f92a3bd6790, timestamp:1727334211402085, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:70, clock:618659840}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277807335692000, v:0}, rec_scn:{val:1726250403931267000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365696606208-2](offset=626688,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170275594572-0](offset=627020,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251879973346-0](offset=627170,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216446493380-0](offset=627396,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a75e40, 0x7f92a3bd6790, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.646515] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=71] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.646523] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.646685] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=16] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:25.646692] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=6] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.646699] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200240}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.646726] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=22] succeed to build merge ctx(tablet_id={id:200240}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200240}, table_count:2, [{i:0, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8637b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625619554, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=27.14ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.646938] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=19] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:25.647024] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:25.647040] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=9] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.647058] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:25.647063] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.647336] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BB-0-0, dag_ret:0, dag_status:2, start_time:1727337625619423, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3460, output_block_cnt_arr:0x7f9287db3468, concurrent_cnt:1, estimate_row_cnt:85417, estimate_occupy_size:134217728, latest_update_ts:1727337625619554, estimated_finish_time:1727337642647399}) [2024-09-26 16:00:25.647347] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=11] succeed to init merge ctx(task={this:0x7f9264d76920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BB-0-0, dag_ret:0, dag_status:2, start_time:1727337625619423, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.647353] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=6] task finish process(ret=0, start_time=1727337625619553, end_time=1727337625647353, runtime=27800, *this={this:0x7f9264d76920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BB-0-0, dag_ret:0, dag_status:2, start_time:1727337625619423, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.647375] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=3] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.647447] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:3, allocator:0x7f926367c7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92664f0320, local_datums:0x7f92664f0320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:25.647491] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=39] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fc060, count:3, allocator:0x7f926367d558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665fc320, local_datums:0x7f92665fc320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:25.647691] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=9][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:25.647736] ERROR issue_dba_error (ob_log.cpp:1875) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.647740] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf2041c8 0xf217c49 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.647747] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:25.647750] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:25.647752] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9262c76108, hex: 313030303030333830, cstr: 100000380},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c760a8, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c760e0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-29413, column_count:20, rowkey_column_count:3, has_column_checksum:0, row_count:100, row_store_type:0, opt:0, var_column_count:0, row_offset:16082, original_length:16422, max_merged_trans_version:1726189672100745000, data_length:16454, data_zlength:5272, data_checksum:4171874375, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92665a6030, buf_size:5272, data_size:16454, row_count:100, column_count:20, max_merged_trans_version:1726189672100745000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:371002229, row_count_delta:22, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16422}) [2024-09-26 16:00:25.647764] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:25.647766] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:25.647767] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:25.647769] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:25.647771] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1526) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, partition_fuser_->get_result_row()={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92664ee148, local_datums:0x7f92664ee148}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee190, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee1c8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3c0, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3f8, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, fuse_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92664f0320, local_datums:0x7f92664f0320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f05d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f926367c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926367c118, limit_param:null, stmt_allocator:0x7f926367c7d8, allocator:0x7f926367c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92664f0320, local_datums:0x7f92664f0320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f05d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665fc320, local_datums:0x7f92665fc320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f926367ce98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926367ce98, limit_param:null, stmt_allocator:0x7f926367d558, allocator:0x7f926367d558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665fc320, local_datums:0x7f92665fc320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337625619319, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:25.647879] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=108][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92664f0320, local_datums:0x7f92664f0320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f05d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f926367c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926367c118, limit_param:null, stmt_allocator:0x7f926367c7d8, allocator:0x7f926367c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92664f0320, local_datums:0x7f92664f0320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f05d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665fc320, local_datums:0x7f92665fc320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f926367ce98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926367ce98, limit_param:null, stmt_allocator:0x7f926367d558, allocator:0x7f926367d558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665fc320, local_datums:0x7f92665fc320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337625619319, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:25.647956] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=76][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:25.647958] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=1] partition merge iter row count(i=0, row_count=23, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.647969] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=9] partition merge iter row count(i=1, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337625619319, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.647979] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:25.647990] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=3][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:25.647994] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:25.647995] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337625647404, end_time=1727337625647995, runtime=591, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BB-0-0, dag_ret:0, dag_status:2, start_time:1727337625619423, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.648003] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BB-0-0, dag_ret:0, dag_status:2, start_time:1727337625619423, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:25.648010] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=28587, dag_cnt=40, dag_cnts_[dag.get_type()]=40, &dag=0x7f9264d76560, dag={ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5BB-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625619423, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.648021] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5BB-0-0] [lt=8] succeed to del sys task(removed_task={start_time:1727337625619416, task_id:YB42C0A8002D-0006230056D8C5BB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.648055] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] succeed to add sys task(task={start_time:1727337625648053, task_id:YB42C0A8002D-0006230056D8C5C5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.648062] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=6] schedule one task(task=0x7f9264d79d00, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.648145] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.648151] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.648153] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.648157] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.648161] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.648167] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.648401] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337625619708, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.648424] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=22] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828761605424, storage_schema:{this:0x7f9264a5f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828761605424, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.648425] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337625619723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.648459] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.648464] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.648459] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=20] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:0, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.648475] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=12] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.648479] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.648485] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:0, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.648470] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264a5f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200004}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1710828761605424, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.648516] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=40] succeed to build merge ctx(tablet_id={id:200004}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828761605424, storage_schema:{this:0x7f9264a5f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200004}, table_count:1, [{i:0, table_key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8621da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625644919, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.55ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.648514] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=22] succeed to build merge ctx(tablet_id={id:1}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:0, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1}, table_count:1, [{i:0, table_key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}, merge_scn:{val:1726241621541056000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625645705, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.78ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.648540] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C2-0-0, dag_ret:0, dag_status:2, start_time:1727337625644905, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264a5f460, output_block_cnt_arr:0x7f9264a5f468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337625644919, estimated_finish_time:1727337642648543}) [2024-09-26 16:00:25.648542] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C3-0-0, dag_ret:0, dag_status:2, start_time:1727337625645641, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}, scanned_row_cnt_arr:0x7f9263a43480, output_block_cnt_arr:0x7f9263a43488, concurrent_cnt:1, estimate_row_cnt:3, estimate_occupy_size:6291456, latest_update_ts:1727337625645705, estimated_finish_time:1727337642648544}) [2024-09-26 16:00:25.648547] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d78d70, type:15, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C2-0-0, dag_ret:0, dag_status:2, start_time:1727337625644905, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.648549] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d792a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C3-0-0, dag_ret:0, dag_status:2, start_time:1727337625645641, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:25.648552] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=5] task finish process(ret=0, start_time=1727337625644917, end_time=1727337625648552, runtime=3635, *this={this:0x7f9264d78d70, type:15, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C2-0-0, dag_ret:0, dag_status:2, start_time:1727337625644905, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.648554] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=5] task finish process(ret=0, start_time=1727337625645703, end_time=1727337625648554, runtime=2851, *this={this:0x7f9264d792a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C3-0-0, dag_ret:0, dag_status:2, start_time:1727337625645641, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:25.648582] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337625619691, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.648590] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337625619741, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.648599] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=16] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:25.648600] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753247280, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828753247280, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.648603] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92728f1480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.648612] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.648614] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.648619] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:412}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753247280, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.648634] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:412}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753247280, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:412}, table_count:1, [{i:0, table_key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625646402, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.21ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.648654] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C4-0-0, dag_ret:0, dag_status:2, start_time:1727337625646383, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a45470, output_block_cnt_arr:0x7f9263a45478, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337625646402, estimated_finish_time:1727337642648656}) [2024-09-26 16:00:25.648659] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d797d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C4-0-0, dag_ret:0, dag_status:2, start_time:1727337625646383, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.648663] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=4] task finish process(ret=0, start_time=1727337625646400, end_time=1727337625648663, runtime=2263, *this={this:0x7f9264d797d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C4-0-0, dag_ret:0, dag_status:2, start_time:1727337625646383, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.648718] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=7] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3223, total_rows=22561) [2024-09-26 16:00:25.648724] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.648729] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847409}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.648756] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=22] succeed to build merge ctx(tablet_id={id:1152921504606847409}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92728f1480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847409}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8640e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625640078, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=8.65ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.648861] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C1-0-0, dag_ret:0, dag_status:2, start_time:1727337625640060, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f15d0, output_block_cnt_arr:0x7f92728f15d8, concurrent_cnt:1, estimate_row_cnt:32439, estimate_occupy_size:18874368, latest_update_ts:1727337625640078, estimated_finish_time:1727337642648870}) [2024-09-26 16:00:25.648867] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d78840, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C1-0-0, dag_ret:0, dag_status:2, start_time:1727337625640060, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.648872] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=5] task finish process(ret=0, start_time=1727337625640076, end_time=1727337625648872, runtime=8796, *this={this:0x7f9264d78840, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C1-0-0, dag_ret:0, dag_status:2, start_time:1727337625640060, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.648890] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337625619666, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.648900] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=9] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:25.648902] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9264a63480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.649010] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=6] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3214, total_rows=22498) [2024-09-26 16:00:25.649014] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.649018] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264a63480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847408}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.649034] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:1152921504606847408}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9264a63480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847408}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625639282, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=9.73ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.649133] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C0-0-0, dag_ret:0, dag_status:2, start_time:1727337625639268, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264a635d0, output_block_cnt_arr:0x7f9264a635d8, concurrent_cnt:1, estimate_row_cnt:35098, estimate_occupy_size:18874368, latest_update_ts:1727337625639282, estimated_finish_time:1727337642649142}) [2024-09-26 16:00:25.649139] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d78310, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C0-0-0, dag_ret:0, dag_status:2, start_time:1727337625639268, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.649144] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=5] task finish process(ret=0, start_time=1727337625639280, end_time=1727337625649144, runtime=9864, *this={this:0x7f9264d78310, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C0-0-0, dag_ret:0, dag_status:2, start_time:1727337625639268, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.649160] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=3] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.649168] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=6] schedule one task(task=0x7f92658b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.649171] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=2] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.649174] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.649178] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=1] schedule one task(task=0x7f9265a043c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.649230] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=200004, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d2e060, count:3, allocator:0x7f92c1d9c7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:25.649272] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=412, range=0x7f92aad52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:7}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d58060, count:4, allocator:0x7f9269ffa7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9264d58350, local_datums:0x7f9264d58350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:25.649299] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:25.649339] ERROR issue_dba_error (ob_log.cpp:1875) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.649342] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.649349] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:25.649351] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:25.649353] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92729ee070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 23, flag: 0, null: 0, ptr: 0x7f92729ee140, hex: 6865617274626561745F736572766963655F65706F6368, cstr: heartbeat_service_epoch},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92729ee0e0, hex: 8808AF9C9C5C0BE8, int: -1726184204453083000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92729ee118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-7627, column_count:7, rowkey_column_count:4, has_column_checksum:0, row_count:1, row_store_type:0, opt:5, var_column_count:0, row_offset:136, original_length:80, max_merged_trans_version:1726184204453083000, data_length:80, data_zlength:80, data_checksum:1549639079, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92b6804090, buf_size:80, data_size:80, row_count:1, column_count:7, max_merged_trans_version:1726184204453083000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:37835160, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:80}) [2024-09-26 16:00:25.649364] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=10][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:25.649366] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=2][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:00:25.649368] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=2][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:00:25.649370] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=1][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:00:25.649371] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=1][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:00:25.649373] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337625619741, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.649386] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=10][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:25.649395] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:25.649398] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:25.649399] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337625649231, end_time=1727337625649399, runtime=168, *this={this:0x7f9265a043c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C4-0-0, dag_ret:0, dag_status:2, start_time:1727337625646383, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.649406] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9265a043c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C4-0-0, dag_ret:0, dag_status:2, start_time:1727337625646383, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:25.649419] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=12][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:25.649436] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=3053, dag_cnt=39, dag_cnts_[dag.get_type()]=39, &dag=0x7f9264d79410, dag={ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C4-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625646383, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.649445] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C4-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337625646377, task_id:YB42C0A8002D-0006230056D8C5C4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.649443] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:8}, cols_desc:{ObIArray:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b0c060, count:5, allocator:0x7f926367c7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b0c360, local_datums:0x7f9265b0c360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:25.649496] ERROR issue_dba_error (ob_log.cpp:1875) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=13][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.649500] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.649506] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:25.649509] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:25.649511] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dce070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dce0a8, hex: C852A9F488570BE8, int: -1726189786434547000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dce0e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19071, column_count:8, rowkey_column_count:3, has_column_checksum:0, row_count:218, row_store_type:0, opt:0, var_column_count:0, row_offset:15549, original_length:16361, max_merged_trans_version:1726190661653507000, data_length:16361, data_zlength:5996, data_checksum:2604580500, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9287dd0030, buf_size:5996, data_size:16361, row_count:218, column_count:8, max_merged_trans_version:1726190661653507000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:83488362, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16361}) [2024-09-26 16:00:25.649521] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=10][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:25.649523] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:25.649524] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:25.649526] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:25.649527] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f92c1d9c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c1d9c118, limit_param:null, stmt_allocator:0x7f92c1d9c7d8, allocator:0x7f92c1d9c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337625619708, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:25.649582] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=55][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:25.649584] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f92c1d9c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c1d9c118, limit_param:null, stmt_allocator:0x7f92c1d9c7d8, allocator:0x7f92c1d9c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337625619708, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:25.649618] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=34][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:25.649620] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=2] partition merge iter row count(i=0, row_count=218, ghost_row_count=0, pkey={tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337625619708, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.649632] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:25.649641] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:25.649644] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:25.649646] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337625649190, end_time=1727337625649646, runtime=456, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C2-0-0, dag_ret:0, dag_status:2, start_time:1727337625644905, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.649651] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=5][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C2-0-0, dag_ret:0, dag_status:2, start_time:1727337625644905, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:25.649658] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=4753, dag_cnt=38, dag_cnts_[dag.get_type()]=38, &dag=0x7f9264d789b0, dag={ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C2-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625644905, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.649666] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C2-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337625644899, task_id:YB42C0A8002D-0006230056D8C5C2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.649727] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=25] multi version scan iterator init succ(param.table_id_=1152921504606847409, range=0x7f92a9952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ec060, count:5, allocator:0x7f9269ffa7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:25.649839] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:25.649882] ERROR issue_dba_error (ob_log.cpp:1875) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.649885] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.649892] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:25.649895] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:25.649896] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 17, flag: 0, null: 0, ptr: 0x7f9264af2178, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264af20a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 15, flag: 0, null: 0, ptr: 0x7f9264af2189, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264af2118, hex: 60D2640CCC290BE8, int: -1726240075808452000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264af2150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-20032, column_count:8, rowkey_column_count:5, has_column_checksum:0, row_count:158, row_store_type:0, opt:0, var_column_count:0, row_offset:15849, original_length:16421, max_merged_trans_version:1726241621541056000, data_length:16421, data_zlength:16421, data_checksum:3599888316, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926f604090, buf_size:16421, data_size:16421, row_count:158, column_count:8, max_merged_trans_version:1726241621541056000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1714289658, row_count_delta:2, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16421}) [2024-09-26 16:00:25.649908] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:25.649910] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:25.649911] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:25.649913] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=2][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:25.649914] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:25.649916] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:25.649934] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=12][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:25.649968] ERROR issue_dba_error (ob_log.cpp:1875) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.649917] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b0c360, local_datums:0x7f9265b0c360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b0c360, local_datums:0x7f9265b0c360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f926367c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926367c118, limit_param:null, stmt_allocator:0x7f926367c7d8, allocator:0x7f926367c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b0c360, local_datums:0x7f9265b0c360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337625619723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:25.649971] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.649976] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=59][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:25.649976] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=5][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:25.649978] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:25.649980] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 7, flag: 0, null: 0, ptr: 0x7f92664f0178, hex: 31303033343330, cstr: 1003430},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f92664f017f, hex: 3230323430393133303435303537, cstr: 20240913045057},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f00e0, hex: 6E48500300000000, int: 55593070},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0118, hex: 58B03BC5A1610BE8, int: -1726178684739801000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-26357, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:250, row_store_type:0, opt:5, var_column_count:0, row_offset:15394, original_length:16334, max_merged_trans_version:1726190643638412000, data_length:16334, data_zlength:5786, data_checksum:2846467531, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9262c6e030, buf_size:5786, data_size:16334, row_count:250, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:796410710, row_count_delta:250, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16334}) [2024-09-26 16:00:25.649991] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:25.649993] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:25.649995] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:25.649996] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:25.649978] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b0c360, local_datums:0x7f9265b0c360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f926367c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926367c118, limit_param:null, stmt_allocator:0x7f926367c7d8, allocator:0x7f926367c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b0c360, local_datums:0x7f9265b0c360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337625619723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:25.650016] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=38][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:25.650019] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=2] partition merge iter row count(i=0, row_count=158, ghost_row_count=0, pkey={tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, table={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337625619723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.650032] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=10][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:25.649997] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9269ffa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269ffa118, limit_param:null, stmt_allocator:0x7f9269ffa7d8, allocator:0x7f9269ffa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337625619691, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:25.650044] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:25.650046] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=48][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:25.650047] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:25.650049] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337625649365, end_time=1727337625650048, runtime=683, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C3-0-0, dag_ret:0, dag_status:2, start_time:1727337625645641, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:25.650056] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C3-0-0, dag_ret:0, dag_status:2, start_time:1727337625645641, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:25.650065] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=4423, dag_cnt=37, dag_cnts_[dag.get_type()]=37, &dag=0x7f9264d78ee0, dag={ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C3-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625645641, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}) [2024-09-26 16:00:25.650072] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C3-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625645632, task_id:YB42C0A8002D-0006230056D8C5C3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.650047] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9269ffa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269ffa118, limit_param:null, stmt_allocator:0x7f9269ffa7d8, allocator:0x7f9269ffa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337625619691, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:25.650080] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=32][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:25.650082] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=1] partition merge iter row count(i=0, row_count=250, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337625619691, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.650093] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:25.650100] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:25.650103] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:25.650104] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337625649690, end_time=1727337625650104, runtime=414, *this={this:0x7f92658b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C1-0-0, dag_ret:0, dag_status:2, start_time:1727337625640060, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.650110] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f92658b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C1-0-0, dag_ret:0, dag_status:2, start_time:1727337625640060, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:25.650106] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337625619765, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.650118] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=10058, dag_cnt=36, dag_cnts_[dag.get_type()]=36, &dag=0x7f9264d78480, dag={ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C1-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625640060, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.650115] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753327496, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753327496, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.650125] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C1-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625640053, task_id:YB42C0A8002D-0006230056D8C5C1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.650128] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.650157] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.650164] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:444}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753327496, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.650187] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1152921504606847408, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685e8060, count:5, allocator:0x7f92636407d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92685e8360, local_datums:0x7f92685e8360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:25.650183] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:444}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753327496, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:444}, table_count:1, [{i:0, table_key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8632d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625648139, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.02ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.650206] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C5-0-0, dag_ret:0, dag_status:2, start_time:1727337625648061, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505490, output_block_cnt_arr:0x7f9266505498, concurrent_cnt:1, estimate_row_cnt:4, estimate_occupy_size:6291456, latest_update_ts:1727337625648139, estimated_finish_time:1727337642650209}) [2024-09-26 16:00:25.650212] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d79d00, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C5-0-0, dag_ret:0, dag_status:2, start_time:1727337625648061, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.650216] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=4] task finish process(ret=0, start_time=1727337625648137, end_time=1727337625650216, runtime=2079, *this={this:0x7f9264d79d00, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C5-0-0, dag_ret:0, dag_status:2, start_time:1727337625648061, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.650406] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=13][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:25.650444] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.650447] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.650453] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:25.650455] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:25.650457] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9262c70178, hex: 30323232362D32312D373825, cstr: 02226-21-78%},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9262c70184, hex: 3230323430393133303435343039, cstr: 20240913045409},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c700e0, hex: CD7F500300000000, int: 55607245},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c70118, hex: E8EC05CCA0600BE8, int: -1726179788432479000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c70150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-11869, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:238, row_store_type:0, opt:5, var_column_count:0, row_offset:15429, original_length:16321, max_merged_trans_version:1726190643638412000, data_length:16321, data_zlength:5388, data_checksum:152456761, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92665a6030, buf_size:5388, data_size:16321, row_count:238, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2114797869, row_count_delta:238, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16321}) [2024-09-26 16:00:25.650467] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=10][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:25.650468] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:25.650470] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:25.650472] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:25.650474] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92685e8360, local_datums:0x7f92685e8360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e83e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92685e8360, local_datums:0x7f92685e8360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e83e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9263640118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263640118, limit_param:null, stmt_allocator:0x7f92636407d8, allocator:0x7f92636407d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92685e8360, local_datums:0x7f92685e8360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e83e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337625619666, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:25.650516] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=42][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:25.650517] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92685e8360, local_datums:0x7f92685e8360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e83e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9263640118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263640118, limit_param:null, stmt_allocator:0x7f92636407d8, allocator:0x7f92636407d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92685e8360, local_datums:0x7f92685e8360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e83e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337625619666, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:25.650563] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=45][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:25.650565] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=1] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337625619666, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.650575] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:25.650583] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:25.650586] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:25.650587] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337625650159, end_time=1727337625650587, runtime=428, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C0-0-0, dag_ret:0, dag_status:2, start_time:1727337625639268, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.650593] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C0-0-0, dag_ret:0, dag_status:2, start_time:1727337625639268, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:25.650599] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=11331, dag_cnt=35, dag_cnts_[dag.get_type()]=35, &dag=0x7f9264d77f50, dag={ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C0-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625639268, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.650606] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C0-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625639260, task_id:YB42C0A8002D-0006230056D8C5C0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.650626] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=2] schedule one task(task=0x7f9287cf83c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.650632] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337625650630, task_id:YB42C0A8002D-0006230056D8C5C6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.650635] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=2] schedule one task(task=0x7f9264d7a230, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.650640] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625650639, task_id:YB42C0A8002D-0006230056D8C5C7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.650643] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=2] schedule one task(task=0x7f9264d7a760, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.650646] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337625650645, task_id:YB42C0A8002D-0006230056D8C5C8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.650648] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=2] schedule one task(task=0x7f9264d7ac90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.650651] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337625650650, task_id:YB42C0A8002D-0006230056D8C5C9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.650653] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=2] schedule one task(task=0x7f9264d7b1c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.650658] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337625650655, task_id:YB42C0A8002D-0006230056D8C5CA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.650660] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=2] schedule one task(task=0x7f9264d7b6f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.650679] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.650683] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.650685] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.650689] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.650689] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=15][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.650693] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.650695] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.650694] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.650699] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.650702] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.650721] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=17][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.650724] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.650725] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.650728] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.650730] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.650734] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241465393532000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.650703] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861cf80, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:528, tablet_addr:[816017-1849-0-2305843670640488448-2](offset=1830912,size=616,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:455}, data_tablet_id:{id:455}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4242650745, row_count:15909}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861d6ec, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305845137371811840-0](offset=1822720,size=1982,type=2,seq=0), type:"MAJOR", tablet_id:{id:455}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389074974000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305845058988660670-0](offset=1824702,size=1909,type=2,seq=0), type:"MINOR", tablet_id:{id:455}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178192070487000, upper_trans_version:1726178192070487000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337625619873, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcec60, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209461810}, this:0x7f92a3bcec60, timestamp:1727334209461810, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:55, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274239678928000, v:0}, rec_scn:{val:1726241818022255001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771548160-2](offset=1826816,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843202489049421-0](offset=1827149,size=180,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881173505-0](offset=1827329,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447693539-0](offset=1827555,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a794f0, 0x7f92a3bcec60, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.650747] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=44] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.650754] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.650771] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=444, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:6, allocator:0x7f92bc9067d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92665f8390, local_datums:0x7f92665f8390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:25.650942] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=11][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:25.650974] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.650976] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.650982] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:25.650984] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:25.650985] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c74070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c740a8, hex: 0300000000000000, int: 3},idx=2:{len: 12, flag: 0, null: 0, ptr: 0x7f9262c741b0, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c74118, hex: 420B000000000000, int: 2882},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c74150, hex: 288E681FE7280BE8, int: -1726241059036951000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c74188, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:24981, column_count:9, rowkey_column_count:6, has_column_checksum:0, row_count:207, row_store_type:0, opt:0, var_column_count:0, row_offset:15610, original_length:16378, max_merged_trans_version:1726241059036951000, data_length:16378, data_zlength:16378, data_checksum:1574137226, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926f404090, buf_size:16378, data_size:16378, row_count:207, column_count:9, max_merged_trans_version:1726241059036951000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:666643265, row_count_delta:3, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16378}) [2024-09-26 16:00:25.650996] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=10][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:25.650997] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=1][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:25.650999] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:25.651000] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=1][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.651003] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=0] partition merge iter row count(i=0, row_count=207, ghost_row_count=0, pkey={tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337625619765, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.651019] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625650743, end_time=1727337625651019, runtime=276, *this={this:0x7f9287cf83c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C5-0-0, dag_ret:0, dag_status:2, start_time:1727337625648061, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.651028] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2966, dag_cnt=34, dag_cnts_[dag.get_type()]=34, &dag=0x7f9264d79940, dag={ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C5-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625648061, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.651034] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C5-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625648053, task_id:YB42C0A8002D-0006230056D8C5C5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.651060] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.651063] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.651065] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.651068] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.651070] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.651075] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.651132] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625651130, task_id:YB42C0A8002D-0006230056D8C5CB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.651135] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=3] schedule one task(task=0x7f9264d7bc20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.651171] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.651171] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.651174] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.651176] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.651177] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.651179] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.651180] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.651184] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.651188] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.651195] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241316899724000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.651183] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8624c80, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:541, tablet_addr:[816017-1849-0-2305843684599267328-2](offset=1966080,size=629,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:492}, data_tablet_id:{id:492}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2528622955, row_count:340306}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8625434, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305850166778642432-0](offset=1949696,size=6666,type=2,seq=0), type:"MAJOR", tablet_id:{id:492}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389176297000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844149529467402-0](offset=1956362,size=1062,type=2,seq=0), type:"MINOR", tablet_id:{id:492}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178192725955000, upper_trans_version:1726178192725955000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337625619893, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd2310, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209496637}, this:0x7f92a3bd2310, timestamp:1727334209496637, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:62, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274246934057000, v:0}, rec_scn:{val:1726241820481609000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697941504-2](offset=1961984,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843215374086476-0](offset=1962316,size=192,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881308684-0](offset=1962508,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447828718-0](offset=1962734,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a78550, 0x7f92a3bd2310, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.651215] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=32] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.651220] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.651375] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337625619873, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.651389] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753332736, storage_schema:{this:0x7f92685e9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=36, max_schema_version_in_memtable=1710828753332736, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.651395] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337625619837, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.651430] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=35] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753357664, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753357664, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.651533] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:25.651537] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.651549] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=14] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:25.651554] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.651544] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:458}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753357664, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.651565] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:458}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753357664, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:458}, table_count:1, [{i:0, table_key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8627b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625650673, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=867us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.651562] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92685e9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:455}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:10, rowkey_column_count:9, schema_rowkey_col_cnt:7, full_stored_col_cnt:38, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753332736, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=26 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.651582] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337625619795, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.651597] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=14] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752918752, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828752918752, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.651595] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=27] succeed to build merge ctx(tablet_id={id:455}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753332736, storage_schema:{this:0x7f92685e9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:455}, table_count:1, [{i:0, table_key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625650683, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=875us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.651612] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.651616] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.651621] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:309}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752918752, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.651640] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:309}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752918752, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:309}, table_count:1, [{i:0, table_key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}, merge_scn:{val:1726241316899724000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8631da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625651164, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=455us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.651663] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C6-0-0, dag_ret:0, dag_status:2, start_time:1727337625650635, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}, scanned_row_cnt_arr:0x7f9265adb490, output_block_cnt_arr:0x7f9265adb498, concurrent_cnt:1, estimate_row_cnt:6, estimate_occupy_size:6291456, latest_update_ts:1727337625651164, estimated_finish_time:1727337642651666}) [2024-09-26 16:00:25.651670] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=18] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:25.651670] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7a230, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C6-0-0, dag_ret:0, dag_status:2, start_time:1727337625650635, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:25.651675] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=5] task finish process(ret=0, start_time=1727337625651162, end_time=1727337625651675, runtime=513, *this={this:0x7f9264d7a230, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C6-0-0, dag_ret:0, dag_status:2, start_time:1727337625650635, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:25.651753] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=20] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:25.651831] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=2] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C8-0-0, dag_ret:0, dag_status:2, start_time:1727337625650648, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a434a0, output_block_cnt_arr:0x7f9263a434a8, concurrent_cnt:1, estimate_row_cnt:14598, estimate_occupy_size:4194304, latest_update_ts:1727337625650673, estimated_finish_time:1727337642651832}) [2024-09-26 16:00:25.651856] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=25] succeed to init merge ctx(task={this:0x7f9264d7ac90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C8-0-0, dag_ret:0, dag_status:2, start_time:1727337625650648, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.651862] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=6] task finish process(ret=0, start_time=1727337625650672, end_time=1727337625651862, runtime=1190, *this={this:0x7f9264d7ac90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C8-0-0, dag_ret:0, dag_status:2, start_time:1727337625650648, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.651913] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337625619815, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.651925] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.651938] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=6] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CA-0-0, dag_ret:0, dag_status:2, start_time:1727337625650660, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685e94b0, output_block_cnt_arr:0x7f92685e94b8, concurrent_cnt:1, estimate_row_cnt:3824, estimate_occupy_size:4194304, latest_update_ts:1727337625650683, estimated_finish_time:1727337642651939}) [2024-09-26 16:00:25.651936] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.651948] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264d7b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CA-0-0, dag_ret:0, dag_status:2, start_time:1727337625650660, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.651951] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=14] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.651953] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=5] task finish process(ret=0, start_time=1727337625650681, end_time=1727337625651953, runtime=1272, *this={this:0x7f9264d7b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CA-0-0, dag_ret:0, dag_status:2, start_time:1727337625650660, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.651957] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:331}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.651975] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:331}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:331}, table_count:1, [{i:0, table_key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}, merge_scn:{val:1726241465393532000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8633ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625650718, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.24ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.651992] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337625619893, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.651998] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C7-0-0, dag_ret:0, dag_status:2, start_time:1727337625650642, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}, scanned_row_cnt_arr:0x7f92685eb480, output_block_cnt_arr:0x7f92685eb488, concurrent_cnt:1, estimate_row_cnt:45, estimate_occupy_size:0, latest_update_ts:1727337625650718, estimated_finish_time:1727337642651998}) [2024-09-26 16:00:25.652004] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7a760, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C7-0-0, dag_ret:0, dag_status:2, start_time:1727337625650642, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:25.652002] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753437040, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=57, max_schema_version_in_memtable=1710828753437040, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.652008] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=4] task finish process(ret=0, start_time=1727337625650717, end_time=1727337625652008, runtime=1291, *this={this:0x7f9264d7a760, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C7-0-0, dag_ret:0, dag_status:2, start_time:1727337625650642, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:25.652024] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337625619854, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.652032] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753417144, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828753417144, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.652058] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1362, total_rows=1362) [2024-09-26 16:00:25.652061] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.652067] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:492}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:12, rowkey_column_count:11, schema_rowkey_col_cnt:9, full_stored_col_cnt:59, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753437040, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=28 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.652101] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=7] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:25.652105] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.652091] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=19] succeed to build merge ctx(tablet_id={id:492}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753437040, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:492}, table_count:1, [{i:0, table_key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8624c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625651167, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=897us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.652109] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:487}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753417144, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.652127] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:487}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753417144, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:487}, table_count:1, [{i:0, table_key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625651056, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.05ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.652162] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CB-0-0, dag_ret:0, dag_status:2, start_time:1727337625651135, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a454c0, output_block_cnt_arr:0x7f9263a454c8, concurrent_cnt:1, estimate_row_cnt:1362, estimate_occupy_size:2097152, latest_update_ts:1727337625651167, estimated_finish_time:1727337642652162}) [2024-09-26 16:00:25.652168] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7bc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CB-0-0, dag_ret:0, dag_status:2, start_time:1727337625651135, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.652173] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=5] task finish process(ret=0, start_time=1727337625651165, end_time=1727337625652173, runtime=1008, *this={this:0x7f9264d7bc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CB-0-0, dag_ret:0, dag_status:2, start_time:1727337625651135, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.652206] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=15] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:25.652300] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C9-0-0, dag_ret:0, dag_status:2, start_time:1727337625650653, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685e74a0, output_block_cnt_arr:0x7f92685e74a8, concurrent_cnt:1, estimate_row_cnt:2199, estimate_occupy_size:0, latest_update_ts:1727337625651056, estimated_finish_time:1727337642652299}) [2024-09-26 16:00:25.652307] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C9-0-0, dag_ret:0, dag_status:2, start_time:1727337625650653, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.652312] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=5] task finish process(ret=0, start_time=1727337625651055, end_time=1727337625652312, runtime=1257, *this={this:0x7f9264d7b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C9-0-0, dag_ret:0, dag_status:2, start_time:1727337625650653, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.652332] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=3] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.652353] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=19] schedule one task(task=0x7f9262cae030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.652356] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=2] schedule one task(task=0x7f9264d84ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.652361] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=2] schedule one task(task=0x7f9264c803c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.652364] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=2] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.652368] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.652426] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=331, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:5, allocator:0x7f92636947d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9263b5a360, local_datums:0x7f9263b5a360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:25.652425] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=309, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:11}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c70060, count:7, allocator:0x7f92864067d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9262c703b0, local_datums:0x7f9262c703b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:25.652588] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.652603] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=455, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:36, schema_rowkey_cnt:7, rowkey_cnt:9, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:7, column_cnt:38}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264dd4060, count:9, allocator:0x7f92c19247d8, capacity:9, init_cnt:9}, datum_utils:{is_oracle_mode:false, rowkey_cnt:9, col_cnt:9, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:38, datum_buffer:{capacity:38, datums:0x7f9264dd5340, local_datums:0x7f9264dd43f0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,]}) [2024-09-26 16:00:25.652624] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.652626] ERROR issue_dba_error (ob_log.cpp:1875) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.652629] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.652640] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=0] partition merge iter row count(i=0, row_count=156, ghost_row_count=0, pkey={tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, table={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337625619795, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.652662] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.652662] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625652380, end_time=1727337625652662, runtime=282, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C6-0-0, dag_ret:0, dag_status:2, start_time:1727337625650635, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:25.652665] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.652672] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2037, dag_cnt=33, dag_cnts_[dag.get_type()]=33, &dag=0x7f9264d79e70, dag={ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C6-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625650635, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}) [2024-09-26 16:00:25.652680] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C6-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625650630, task_id:YB42C0A8002D-0006230056D8C5C6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.652676] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=0] partition merge iter row count(i=0, row_count=196, ghost_row_count=0, pkey={tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, table={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337625619815, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.652696] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625652372, end_time=1727337625652696, runtime=324, *this={this:0x7f9262cae030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C7-0-0, dag_ret:0, dag_status:2, start_time:1727337625650642, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:25.652705] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2063, dag_cnt=32, dag_cnts_[dag.get_type()]=32, &dag=0x7f9264d7a3a0, dag={ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C7-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625650642, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}) [2024-09-26 16:00:25.652712] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C7-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625650639, task_id:YB42C0A8002D-0006230056D8C5C7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.652732] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=487, range=0x7f92aaad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:11}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d58060, count:8, allocator:0x7f9275dfe7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9264d583e0, local_datums:0x7f9264d583e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:25.652762] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=458, range=0x7f92a9cd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fc060, count:8, allocator:0x7f92bc9047d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92665fc3e0, local_datums:0x7f92665fc3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:25.652844] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=19][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.652881] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.652884] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.652894] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=1] partition merge iter row count(i=0, row_count=140, ghost_row_count=0, pkey={tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337625619873, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.652917] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625652557, end_time=1727337625652917, runtime=360, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CA-0-0, dag_ret:0, dag_status:2, start_time:1727337625650660, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.652927] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2267, dag_cnt=31, dag_cnts_[dag.get_type()]=31, &dag=0x7f9264d7b330, dag={ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CA-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625650660, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.652934] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CA-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625650655, task_id:YB42C0A8002D-0006230056D8C5CA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.652936] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.652964] ERROR issue_dba_error (ob_log.cpp:1875) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.652967] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.652966] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.652974] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=0] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337625619854, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.652990] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625652704, end_time=1727337625652990, runtime=286, *this={this:0x7f9264c803c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C9-0-0, dag_ret:0, dag_status:2, start_time:1727337625650653, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.652996] ERROR issue_dba_error (ob_log.cpp:1875) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.652998] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2345, dag_cnt=30, dag_cnts_[dag.get_type()]=30, &dag=0x7f9264d7ae00, dag={ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C9-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625650653, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.652998] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.653004] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C9-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625650650, task_id:YB42C0A8002D-0006230056D8C5C9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.652991] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=492, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:57, schema_rowkey_cnt:9, rowkey_cnt:11, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:9, column_cnt:59}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92729fa060, count:11, allocator:0x7f92865547d8, capacity:11, init_cnt:11}, datum_utils:{is_oracle_mode:false, rowkey_cnt:11, col_cnt:11, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:59, datum_buffer:{capacity:59, datums:0x7f92728fa050, local_datums:0x7f92729fa440}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,col_id=43:NOP,col_id=44:NOP,col_id=45:NOP,col_id=46:NOP,col_id=47:NOP,col_id=48:NOP,col_id=49:NOP,col_id=50:NOP,col_id=51:NOP,col_id=52:NOP,col_id=53:NOP,col_id=54:NOP,col_id=55:NOP,col_id=56:NOP,col_id=57:NOP,col_id=58:NOP,]}) [2024-09-26 16:00:25.653006] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=0] partition merge iter row count(i=0, row_count=247, ghost_row_count=0, pkey={tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337625619837, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.653021] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625653019, task_id:YB42C0A8002D-0006230056D8C5CC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.653024] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=3] schedule one task(task=0x7f9264d7c150, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.653022] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625652736, end_time=1727337625653022, runtime=286, *this={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C8-0-0, dag_ret:0, dag_status:2, start_time:1727337625650648, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.653031] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337625653030, task_id:YB42C0A8002D-0006230056D8C5CD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.653034] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=2] schedule one task(task=0x7f9264d7c680, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.653045] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337625653045, task_id:YB42C0A8002D-0006230056D8C5CE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.653048] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=2] schedule one task(task=0x7f9264d7cbb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.653053] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337625653052, task_id:YB42C0A8002D-0006230056D8C5CF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.653055] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=2] schedule one task(task=0x7f9264d7d0e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.653079] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.653084] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.653086] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726077706339704027) [2024-09-26 16:00:25.653087] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.653090] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.653091] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.653093] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.653094] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.653097] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.653101] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.653100] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.653107] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.653123] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.653126] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.653127] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.653130] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.653132] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.653136] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.653152] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2504, dag_cnt=29, dag_cnts_[dag.get_type()]=29, &dag=0x7f9264d7a8d0, dag={ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5C8-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625650648, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.653160] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5C8-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337625650645, task_id:YB42C0A8002D-0006230056D8C5C8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.653182] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337625653180, task_id:YB42C0A8002D-0006230056D8C5D0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.653185] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=3] schedule one task(task=0x7f9264d7d610, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.653200] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.653203] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.653205] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.653208] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.653212] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=18][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.653245] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.653210] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8641da0, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:487, tablet_addr:[816017-1849-0-2305843626617249792-2](offset=2007040,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101009}, data_tablet_id:{id:140}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726073917016589000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3251381279, row_count:150}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8642458, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844188184215552-0](offset=1998848,size=1098,type=2,seq=0), type:"MAJOR", tablet_id:{id:101009}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726070403667750001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843957329724490-0](offset=1999946,size=883,type=2,seq=0), type:"MINOR", tablet_id:{id:101009}, scn_range:{start_scn:{val:1726073917016589000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726167395874925000, upper_trans_version:1726167395874925000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337625619987, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd0ba0, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495816}, this:0x7f92a3bd0ba0, timestamp:1727334209495816, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:59, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726256203821911000, v:0}, rec_scn:{val:1726241803398655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697982464-2](offset=2002944,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276970828-0](offset=2003276,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881349602-0](offset=2003426,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7bc00, 0x7f92a3bd0ba0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.653248] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.653253] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=42] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.653258] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.653257] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=0] partition merge iter row count(i=0, row_count=61, ghost_row_count=0, pkey={tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337625619893, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.653286] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=7] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:25.653290] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625652959, end_time=1727337625653289, runtime=330, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CB-0-0, dag_ret:0, dag_status:2, start_time:1727337625651135, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.653300] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2164, dag_cnt=28, dag_cnts_[dag.get_type()]=28, &dag=0x7f9264d7b860, dag={ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CB-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625651135, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.653307] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CB-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625651130, task_id:YB42C0A8002D-0006230056D8C5CB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.653340] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.653344] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.653346] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.653350] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.653353] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.653364] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625653363, task_id:YB42C0A8002D-0006230056D8C5D1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.653368] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=4] schedule one task(task=0x7f9264d7db40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.653368] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.653393] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.653396] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.653398] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:25.653402] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.653404] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.653409] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.653430] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337625619910, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.653440] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753441792, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828753441792, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.653517] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1295, total_rows=1295) [2024-09-26 16:00:25.653464] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337625619987, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.653523] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.653522] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=58] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.653535] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.653539] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.653529] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:501}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753441792, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.653544] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101009}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:25.653548] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:501}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753441792, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:501}, table_count:1, [{i:0, table_key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625653120, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=407us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.653561] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101009}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101009}, table_count:1, [{i:0, table_key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8641da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625653197, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=345us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.653584] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D0-0-0, dag_ret:0, dag_status:2, start_time:1727337625653185, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab470, output_block_cnt_arr:0x7f92684ab478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337625653197, estimated_finish_time:1727337642653584}) [2024-09-26 16:00:25.653590] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7d610, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D0-0-0, dag_ret:0, dag_status:2, start_time:1727337625653185, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.653595] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=5] task finish process(ret=0, start_time=1727337625653195, end_time=1727337625653595, runtime=400, *this={this:0x7f9264d7d610, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D0-0-0, dag_ret:0, dag_status:2, start_time:1727337625653185, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.653621] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CC-0-0, dag_ret:0, dag_status:2, start_time:1727337625653024, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505490, output_block_cnt_arr:0x7f9266505498, concurrent_cnt:1, estimate_row_cnt:1295, estimate_occupy_size:0, latest_update_ts:1727337625653120, estimated_finish_time:1727337642653620}) [2024-09-26 16:00:25.653619] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=11] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337625619971, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.653626] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7c150, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CC-0-0, dag_ret:0, dag_status:2, start_time:1727337625653024, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.653628] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.653631] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=5] task finish process(ret=0, start_time=1727337625653119, end_time=1727337625653631, runtime=512, *this={this:0x7f9264d7c150, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CC-0-0, dag_ret:0, dag_status:2, start_time:1727337625653024, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.653635] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=5] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.653638] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.653642] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:140}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.653654] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=9] succeed to build merge ctx(tablet_id={id:140}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:140}, table_count:1, [{i:0, table_key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8622d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625653081, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=558us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.653672] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=12] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CF-0-0, dag_ret:0, dag_status:2, start_time:1727337625653055, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b23460, output_block_cnt_arr:0x7f9264b23468, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337625653081, estimated_finish_time:1727337642653672}) [2024-09-26 16:00:25.653677] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CF-0-0, dag_ret:0, dag_status:2, start_time:1727337625653055, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.653681] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=4] task finish process(ret=0, start_time=1727337625653079, end_time=1727337625653681, runtime=602, *this={this:0x7f9264d7d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CF-0-0, dag_ret:0, dag_status:2, start_time:1727337625653055, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.653706] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=2] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.653711] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=3] schedule one task(task=0x7f9264d85260, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.653715] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=1] schedule one task(task=0x7f9264d84ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.653744] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=501, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fc060, count:6, allocator:0x7f92864e07d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92665fc390, local_datums:0x7f92665fc390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:25.653812] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.653857] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.653861] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.653870] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=0] partition merge iter row count(i=0, row_count=56, ghost_row_count=0, pkey={tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337625619910, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.653889] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625653718, end_time=1727337625653889, runtime=171, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CC-0-0, dag_ret:0, dag_status:2, start_time:1727337625653024, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.653897] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=873, dag_cnt=27, dag_cnts_[dag.get_type()]=27, &dag=0x7f9264d7bd90, dag={ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CC-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625653024, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.653904] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CC-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625653019, task_id:YB42C0A8002D-0006230056D8C5CC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.653918] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625653916, task_id:YB42C0A8002D-0006230056D8C5D2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.653922] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=3] schedule one task(task=0x7f9264d7e070, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.653934] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=11] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337625620008, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.653943] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.653953] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.653956] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.653961] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101010}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:25.653976] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:101010}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101010}, table_count:1, [{i:0, table_key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8642d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625653381, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=578us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.653996] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D1-0-0, dag_ret:0, dag_status:2, start_time:1727337625653368, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f470, output_block_cnt_arr:0x7f9264b1f478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337625653381, estimated_finish_time:1727337642653995}) [2024-09-26 16:00:25.654001] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7db40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D1-0-0, dag_ret:0, dag_status:2, start_time:1727337625653368, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.654006] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=4] task finish process(ret=0, start_time=1727337625653379, end_time=1727337625654005, runtime=626, *this={this:0x7f9264d7db40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D1-0-0, dag_ret:0, dag_status:2, start_time:1727337625653368, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.654024] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=1] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.654038] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=7][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.654041] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.654043] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:25.654047] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.654050] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.654055] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.654074] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101010, range=0x7f92ab0d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:4, allocator:0x7f92c4a5e7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263b5a350, local_datums:0x7f9263b5a350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:25.654110] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.654146] ERROR issue_dba_error (ob_log.cpp:1875) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.654149] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.654158] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337625620008, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.654209] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625654035, end_time=1727337625654177, runtime=142, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D1-0-0, dag_ret:0, dag_status:2, start_time:1727337625653368, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.654221] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=853, dag_cnt=26, dag_cnts_[dag.get_type()]=26, &dag=0x7f9264d7d780, dag={ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D1-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625653368, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.654228] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D1-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625653363, task_id:YB42C0A8002D-0006230056D8C5D1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.654278] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625654277, task_id:YB42C0A8002D-0006230056D8C5D3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.654282] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=3] schedule one task(task=0x7f9264d7e5a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.654305] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.654309] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.654311] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:25.654315] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.654318] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864df20, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:503, tablet_addr:[814042-5586-0-2305843643796283392-2](offset=1171456,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:377}, data_tablet_id:{id:377}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4164021307, row_count:17}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864e5fc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844760487772160-0](offset=1163264,size=1631,type=2,seq=0), type:"MAJOR", tablet_id:{id:377}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563080818000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843989541144159-0](offset=1164895,size=913,type=2,seq=0), type:"MINI", tablet_id:{id:377}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557779816000, upper_trans_version:1726121557779816000}, {i:1, addr:[814042-5586-0-2305843969140050416-0](offset=1165808,size=894,type=2,seq=0), type:"MINI", tablet_id:{id:377}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986451466000, upper_trans_version:1726149986451466000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337625620046, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdb5b0, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220773832}, this:0x7f92a3bdb5b0, timestamp:1727334220773832, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:80, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293334630000, v:0}, rec_scn:{val:1726264800181268000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513780224-2](offset=1167360,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843176718586294-0](offset=1167798,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880514130-0](offset=1167954,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447034164-0](offset=1168180,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7e310, 0x7f92a3bdb5b0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.654358] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=40] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.654364] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.654476] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337625620028, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.654484] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752949376, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=20, max_schema_version_in_memtable=1710828752949376, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.654500] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.654503] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.654507] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:323}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:22, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752949376, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.654523] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:323}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752949376, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:323}, table_count:1, [{i:0, table_key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625654033, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=473us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.654548] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D2-0-0, dag_ret:0, dag_status:2, start_time:1727337625653921, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db9480, output_block_cnt_arr:0x7f9287db9488, concurrent_cnt:1, estimate_row_cnt:205, estimate_occupy_size:0, latest_update_ts:1727337625654033, estimated_finish_time:1727337642654548}) [2024-09-26 16:00:25.654553] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7e070, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D2-0-0, dag_ret:0, dag_status:2, start_time:1727337625653921, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.654558] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=5] task finish process(ret=0, start_time=1727337625654032, end_time=1727337625654558, runtime=526, *this={this:0x7f9264d7e070, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D2-0-0, dag_ret:0, dag_status:2, start_time:1727337625653921, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.654567] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.654636] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=323, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:20, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:22}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:5, allocator:0x7f92bc9047d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,]}) [2024-09-26 16:00:25.654856] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:25.654892] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.654895] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.654901] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:25.654903] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:25.654905] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e070, hex: 0000000000000000, int: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e0a8, hex: 9B00000000000000, int: 155},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e0e0, hex: 1300000000000000, int: 19},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e118, hex: B00AB6F30C2D0BE8, int: -1726236498514802000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e150, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-13066, column_count:22, rowkey_column_count:5, has_column_checksum:0, row_count:171, row_store_type:0, opt:0, var_column_count:0, row_offset:15755, original_length:16379, max_merged_trans_version:1726236508791484000, data_length:16379, data_zlength:16379, data_checksum:241358913, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926f204090, buf_size:16379, data_size:16379, row_count:171, column_count:22, max_merged_trans_version:1726236508791484000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:637720480, row_count_delta:45, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16379}) [2024-09-26 16:00:25.654917] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:25.654919] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:25.654920] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:25.654921] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:25.654923] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:25.654924] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:25.654926] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726236498514802000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a8, hex: 9B00000000000000, int: 155},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3e0, hex: 1300000000000000, int: 19},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a418, hex: B00AB6F30C2D0BE8, int: -1726236498514802000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a450, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9275bbc45f, hex: 89E3D7E303140600, int: 1710856800297865},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9275bbc467, hex: A4E83FC100220600, int: 1726236497799332},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a4f8, hex: A001000000000000, int: 416},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a568, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5a0, hex: 0000000000000000, int: 0},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a610, hex: 0000000000000000, int: 0},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a648, hex: 0000000000000000, int: 0},col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a680, hex: 0000000000000000, int: 0},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:null,col_id=19:null,col_id=20:null,col_id=21:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a808, hex: 0100000000000000, int: 1, num_digit0: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:323}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752949376, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726236498514802000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a8, hex: 9B00000000000000, int: 155},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3e0, hex: 1300000000000000, int: 19},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a418, hex: B00AB6F30C2D0BE8, int: -1726236498514802000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a450, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9275bbc45f, hex: 89E3D7E303140600, int: 1710856800297865},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9275bbc467, hex: A4E83FC100220600, int: 1726236497799332},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a4f8, hex: A001000000000000, int: 416},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a568, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5a0, hex: 0000000000000000, int: 0},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a610, hex: 0000000000000000, int: 0},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a648, hex: 0000000000000000, int: 0},col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a680, hex: 0000000000000000, int: 0},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:null,col_id=19:null,col_id=20:null,col_id=21:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a808, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f92bc904118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92bc904118, limit_param:null, stmt_allocator:0x7f92bc9047d8, allocator:0x7f92bc9047d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726236498514802000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a8, hex: 9B00000000000000, int: 155},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3e0, hex: 1300000000000000, int: 19},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a418, hex: B00AB6F30C2D0BE8, int: -1726236498514802000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a450, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9275bbc45f, hex: 89E3D7E303140600, int: 1710856800297865},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9275bbc467, hex: A4E83FC100220600, int: 1726236497799332},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a4f8, hex: A001000000000000, int: 416},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a568, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5a0, hex: 0000000000000000, int: 0},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a610, hex: 0000000000000000, int: 0},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a648, hex: 0000000000000000, int: 0},col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a680, hex: 0000000000000000, int: 0},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:null,col_id=19:null,col_id=20:null,col_id=21:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a808, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":0},{"BIGINT":155},{"BIGINT":19}, value_iter:{value:{this=0x7f92751719c0 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f9275bbc3e8 latest_compact_node=(nil) max_trans_version={val:1726236498514802000, v:0} max_trans_id=125543362 max_elr_trans_version={val:1726236498514802000, v:0} max_elr_trans_id=125543362 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=6 max_modify_scn={val:1726236498514802000, v:0} min_modify_scn={val:1726199007654744001, v:0}}, version_iter:this=0x7f9275bb9708 trans_version={val:1726236498514802000, v:0} scn={val:1726236498514802000, v:0} tx_id={txid:125543362} prev=0x7f9275bb0f50 next=0x7f9275bbc3e8 modify_count=2 acc_checksum=3980153904 version=1726236009041170 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:52} seq_no=1726236497797898, multi_version_iter:this=0x7f9275bbc3e8 trans_version={val:1726236498514802000, v:0} scn={val:1726236498514802000, v:0} tx_id={txid:125543362} prev=0x7f9275bb9708 next=(nil) modify_count=3 acc_checksum=1196082307 version=1726236009041170 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:78} seq_no=1726236497989235, max_committed_trans_version:1726236498514802000, cur_trans_version:{val:1726236498514802000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:171, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337625620028, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:22, cur_pos:0, count:0}}) [2024-09-26 16:00:25.654991] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=65][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:25.654993] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:323}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752949376, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726236498514802000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a8, hex: 9B00000000000000, int: 155},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3e0, hex: 1300000000000000, int: 19},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a418, hex: B00AB6F30C2D0BE8, int: -1726236498514802000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a450, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9275bbc45f, hex: 89E3D7E303140600, int: 1710856800297865},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9275bbc467, hex: A4E83FC100220600, int: 1726236497799332},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a4f8, hex: A001000000000000, int: 416},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a568, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5a0, hex: 0000000000000000, int: 0},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a610, hex: 0000000000000000, int: 0},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a648, hex: 0000000000000000, int: 0},col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a680, hex: 0000000000000000, int: 0},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:null,col_id=19:null,col_id=20:null,col_id=21:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a808, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f92bc904118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92bc904118, limit_param:null, stmt_allocator:0x7f92bc9047d8, allocator:0x7f92bc9047d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726236498514802000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a8, hex: 9B00000000000000, int: 155},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3e0, hex: 1300000000000000, int: 19},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a418, hex: B00AB6F30C2D0BE8, int: -1726236498514802000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a450, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9275bbc45f, hex: 89E3D7E303140600, int: 1710856800297865},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9275bbc467, hex: A4E83FC100220600, int: 1726236497799332},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a4f8, hex: A001000000000000, int: 416},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a568, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5a0, hex: 0000000000000000, int: 0},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a610, hex: 0000000000000000, int: 0},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a648, hex: 0000000000000000, int: 0},col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a680, hex: 0000000000000000, int: 0},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:null,col_id=19:null,col_id=20:null,col_id=21:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a808, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":0},{"BIGINT":155},{"BIGINT":19}, value_iter:{value:{this=0x7f92751719c0 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f9275bbc3e8 latest_compact_node=(nil) max_trans_version={val:1726236498514802000, v:0} max_trans_id=125543362 max_elr_trans_version={val:1726236498514802000, v:0} max_elr_trans_id=125543362 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=6 max_modify_scn={val:1726236498514802000, v:0} min_modify_scn={val:1726199007654744001, v:0}}, version_iter:this=0x7f9275bb9708 trans_version={val:1726236498514802000, v:0} scn={val:1726236498514802000, v:0} tx_id={txid:125543362} prev=0x7f9275bb0f50 next=0x7f9275bbc3e8 modify_count=2 acc_checksum=3980153904 version=1726236009041170 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:52} seq_no=1726236497797898, multi_version_iter:this=0x7f9275bbc3e8 trans_version={val:1726236498514802000, v:0} scn={val:1726236498514802000, v:0} tx_id={txid:125543362} prev=0x7f9275bb9708 next=(nil) modify_count=3 acc_checksum=1196082307 version=1726236009041170 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:78} seq_no=1726236497989235, max_committed_trans_version:1726236498514802000, cur_trans_version:{val:1726236498514802000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:171, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337625620028, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:22, cur_pos:0, count:0}}]) [2024-09-26 16:00:25.655033] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=40][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:25.655035] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2] partition merge iter row count(i=0, row_count=171, ghost_row_count=0, pkey={tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337625620028, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.655048] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=10][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:25.655056] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:25.655059] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:25.655061] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337625654608, end_time=1727337625655061, runtime=453, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D2-0-0, dag_ret:0, dag_status:2, start_time:1727337625653921, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.655052] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=140, range=0x7f92a9a56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:3, allocator:0x7f92636447d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:25.655068] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D2-0-0, dag_ret:0, dag_status:2, start_time:1727337625653921, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:25.655077] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=6] dag finished(dag_ret=-4184, runtime=1155, dag_cnt=25, dag_cnts_[dag.get_type()]=25, &dag=0x7f9264d7dcb0, dag={ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D2-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625653921, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.655084] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D2-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625653916, task_id:YB42C0A8002D-0006230056D8C5D2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.655108] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=28] succeed to add sys task(task={start_time:1727337625655107, task_id:YB42C0A8002D-0006230056D8C5D4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.655112] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=3] schedule one task(task=0x7f9264d7ead0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.655111] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=18][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.655130] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.655133] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.655135] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:25.655140] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.655149] ERROR issue_dba_error (ob_log.cpp:1875) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.655153] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.655163] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337625619971, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.655143] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864eec0, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[814042-5586-0-2305843650238771200-2](offset=1208320,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:297}, data_tablet_id:{id:297}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726063409221604000, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2361333062, row_count:24}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844531780800512-0](offset=1200128,size=1418,type=2,seq=0), type:"MAJOR", tablet_id:{id:297}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063202518238000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843919747962250-0](offset=1201546,size=848,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726063409221604000, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557514739000, upper_trans_version:1726121557514739000}, {i:1, addr:[814042-5586-0-2305843920821704922-0](offset=1202394,size=849,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149603136610000, upper_trans_version:1726149603136610000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337625620063, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd86d0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220478269}, this:0x7f92a3bd86d0, timestamp:1727334220478269, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:74, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726264804632655000, v:0}, rec_scn:{val:1726264804632655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513817088-2](offset=1204224,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843183161074102-0](offset=1204662,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880551000-0](offset=1204824,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447071034-0](offset=1205050,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7eae0, 0x7f92a3bd86d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.655183] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=40] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.655189] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.655185] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625655019, end_time=1727337625655185, runtime=166, *this={this:0x7f9264d85260, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CF-0-0, dag_ret:0, dag_status:2, start_time:1727337625653055, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.655196] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2141, dag_cnt=24, dag_cnts_[dag.get_type()]=24, &dag=0x7f9264d7cd20, dag={ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CF-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625653055, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.655206] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CF-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337625653052, task_id:YB42C0A8002D-0006230056D8C5CF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.655237] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337625619932, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.655243] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337625620046, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.655248] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753360568, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753360568, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.655252] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=41, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.655259] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.655262] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.655264] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.655267] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.655280] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:377}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:43, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.655267] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:472}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753360568, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.655300] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:377}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:377}, table_count:1, [{i:0, table_key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625654294, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=976us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.655322] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D3-0-0, dag_ret:0, dag_status:2, start_time:1727337625654282, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f480, output_block_cnt_arr:0x7f9264b1f488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337625654294, estimated_finish_time:1727337642655322}) [2024-09-26 16:00:25.655310] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=38] succeed to build merge ctx(tablet_id={id:472}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753360568, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:472}, table_count:1, [{i:0, table_key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625653334, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.93ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.655328] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D3-0-0, dag_ret:0, dag_status:2, start_time:1727337625654282, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.655333] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=5] task finish process(ret=0, start_time=1727337625654292, end_time=1727337625655333, runtime=1041, *this={this:0x7f9264d7e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D3-0-0, dag_ret:0, dag_status:2, start_time:1727337625654282, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.655335] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CD-0-0, dag_ret:0, dag_status:2, start_time:1727337625653033, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1460, output_block_cnt_arr:0x7f92728f1468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337625653334, estimated_finish_time:1727337642655334}) [2024-09-26 16:00:25.655341] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7c680, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CD-0-0, dag_ret:0, dag_status:2, start_time:1727337625653033, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.655346] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=5] task finish process(ret=0, start_time=1727337625653332, end_time=1727337625655346, runtime=2014, *this={this:0x7f9264d7c680, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CD-0-0, dag_ret:0, dag_status:2, start_time:1727337625653033, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.655349] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=2] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.655355] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337625655354, task_id:YB42C0A8002D-0006230056D8C5D5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.655358] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=3] schedule one task(task=0x7f9264d7f000, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.655449] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=377, range=0x7f92a98d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:41, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:43}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265a76060, count:5, allocator:0x7f9275dfe7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:43, datum_buffer:{capacity:43, datums:0x7f9265a772b0, local_datums:0x7f9265a76360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,]}) [2024-09-26 16:00:25.655473] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=101009, range=0x7f92a9b52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dce060, count:4, allocator:0x7f92c19447d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287dce350, local_datums:0x7f9287dce350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:25.655491] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.655512] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.655526] ERROR issue_dba_error (ob_log.cpp:1875) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.655529] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.655539] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=0] partition merge iter row count(i=0, row_count=6, ghost_row_count=0, pkey={tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337625620046, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.655547] ERROR issue_dba_error (ob_log.cpp:1875) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.655550] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.655557] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625655370, end_time=1727337625655557, runtime=187, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D3-0-0, dag_ret:0, dag_status:2, start_time:1727337625654282, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.655560] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337625619987, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.655566] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1284, dag_cnt=23, dag_cnts_[dag.get_type()]=23, &dag=0x7f9264d7e1e0, dag={ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D3-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625654282, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.655575] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D3-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337625654277, task_id:YB42C0A8002D-0006230056D8C5D3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.655579] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625655429, end_time=1727337625655579, runtime=150, *this={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D0-0-0, dag_ret:0, dag_status:2, start_time:1727337625653185, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.655589] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2404, dag_cnt=22, dag_cnts_[dag.get_type()]=22, &dag=0x7f9264d7d250, dag={ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D0-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625653185, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.655597] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D0-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625653180, task_id:YB42C0A8002D-0006230056D8C5D0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.655603] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.655607] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.655609] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:25.655612] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.655615] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.655652] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.655685] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=2] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.655691] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337625655689, task_id:YB42C0A8002D-0006230056D8C5D6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.655694] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=2] schedule one task(task=0x7f9264d7f530, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.655699] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337625655698, task_id:YB42C0A8002D-0006230056D8C5D7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.655701] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=2] schedule one task(task=0x7f9264d7fa60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.655716] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.655718] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.655720] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:25.655723] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.655725] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.655729] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.655738] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=472, range=0x7f92aa352200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d30060, count:3, allocator:0x7f9269ffa7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f9287d30320, local_datums:0x7f9287d30320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:25.655749] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337625620063, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.655758] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=16, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.655769] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.655773] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.655775] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.655781] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:297}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:18, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.655807] ERROR issue_dba_error (ob_log.cpp:1875) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.655809] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.655816] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=0] partition merge iter row count(i=0, row_count=17, ghost_row_count=0, pkey={tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337625619932, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.655811] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:297}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:297}, table_count:1, [{i:0, table_key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625655125, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=653us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.655833] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337625655697, end_time=1727337625655833, runtime=136, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CD-0-0, dag_ret:0, dag_status:2, start_time:1727337625653033, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.655841] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2808, dag_cnt=21, dag_cnts_[dag.get_type()]=21, &dag=0x7f9264d7c2c0, dag={ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CD-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625653033, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.655849] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CD-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625653030, task_id:YB42C0A8002D-0006230056D8C5CD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.655832] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D4-0-0, dag_ret:0, dag_status:2, start_time:1727337625655111, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3490, output_block_cnt_arr:0x7f92728f3498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337625655125, estimated_finish_time:1727337642655832}) [2024-09-26 16:00:25.655874] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=41] succeed to init merge ctx(task={this:0x7f9264d7ead0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D4-0-0, dag_ret:0, dag_status:2, start_time:1727337625655111, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.655878] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.655881] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.655883] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:25.655880] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=5] task finish process(ret=0, start_time=1727337625655123, end_time=1727337625655879, runtime=756, *this={this:0x7f9264d7ead0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D4-0-0, dag_ret:0, dag_status:2, start_time:1727337625655111, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.655886] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.655889] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.655895] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.655897] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.655916] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337625655914, task_id:YB42C0A8002D-0006230056D8C5D8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.655919] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=3] schedule one task(task=0x7f9264d7ff90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.655959] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=297, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:16, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:18}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:6, allocator:0x7f92865b67d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:18, datum_buffer:{capacity:32, datums:0x7f9264a62390, local_datums:0x7f9264a62390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,]}) [2024-09-26 16:00:25.656011] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=37][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.656046] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.656049] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.656057] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337625620063, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.656075] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625655930, end_time=1727337625656074, runtime=144, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D4-0-0, dag_ret:0, dag_status:2, start_time:1727337625655111, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.656083] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=972, dag_cnt=20, dag_cnts_[dag.get_type()]=20, &dag=0x7f9264d7e710, dag={ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D4-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625655111, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.656090] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D4-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625655107, task_id:YB42C0A8002D-0006230056D8C5D4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.656134] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337625620133, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.656142] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.656153] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.656156] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.656162] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101066}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.656179] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:101066}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101066}, table_count:1, [{i:0, table_key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8653ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625655712, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=447us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.656207] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D7-0-0, dag_ret:0, dag_status:2, start_time:1727337625655701, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505490, output_block_cnt_arr:0x7f9266505498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337625655712, estimated_finish_time:1727337642656206}) [2024-09-26 16:00:25.656212] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7fa60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D7-0-0, dag_ret:0, dag_status:2, start_time:1727337625655701, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.656217] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=5] task finish process(ret=0, start_time=1727337625655711, end_time=1727337625656217, runtime=506, *this={this:0x7f9264d7fa60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D7-0-0, dag_ret:0, dag_status:2, start_time:1727337625655701, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.656277] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337625620085, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.656286] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.656283] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337625620105, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.656296] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.656298] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.656293] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.656311] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.656303] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101065}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.656313] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.656318] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:101065}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101065}, table_count:1, [{i:0, table_key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625655598, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=703us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.656319] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:333}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.656338] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D5-0-0, dag_ret:0, dag_status:2, start_time:1727337625655358, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515490, output_block_cnt_arr:0x7f9268515498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337625655598, estimated_finish_time:1727337642656337}) [2024-09-26 16:00:25.656343] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7f000, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D5-0-0, dag_ret:0, dag_status:2, start_time:1727337625655358, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.656347] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=4] task finish process(ret=0, start_time=1727337625655596, end_time=1727337625656347, runtime=751, *this={this:0x7f9264d7f000, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D5-0-0, dag_ret:0, dag_status:2, start_time:1727337625655358, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.656338] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:333}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:333}, table_count:1, [{i:0, table_key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8651da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625655874, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=442us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.656359] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=2] schedule one task(task=0x7f926652a3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.656364] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=2] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.656369] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625656367, task_id:YB42C0A8002D-0006230056D8C5D9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.656366] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D6-0-0, dag_ret:0, dag_status:2, start_time:1727337625655693, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f490, output_block_cnt_arr:0x7f9264b1f498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337625655874, estimated_finish_time:1727337642656366}) [2024-09-26 16:00:25.656372] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=2] schedule one task(task=0x7f9264d804c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.656372] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7f530, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D6-0-0, dag_ret:0, dag_status:2, start_time:1727337625655693, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.656377] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=5] task finish process(ret=0, start_time=1727337625655872, end_time=1727337625656377, runtime=505, *this={this:0x7f9264d7f530, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D6-0-0, dag_ret:0, dag_status:2, start_time:1727337625655693, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.656394] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.656398] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.656399] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:25.656403] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.656406] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.656412] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.656475] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101066, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:7}, cols_desc:{ObIArray:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b58060, count:7, allocator:0x7f92636887d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9263b583b0, local_datums:0x7f9263b583b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:25.656528] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337625619950, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.656537] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753001832, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=21, max_schema_version_in_memtable=1710828753001832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.656547] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.656550] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.656554] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:342}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:23, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753001832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.656570] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:342}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753001832, storage_schema:{this:0x7f9265adb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:342}, table_count:1, [{i:0, table_key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625653066, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.49ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.656606] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=31] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CE-0-0, dag_ret:0, dag_status:2, start_time:1727337625653048, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265adb490, output_block_cnt_arr:0x7f9265adb498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337625653066, estimated_finish_time:1727337642656605}) [2024-09-26 16:00:25.656611] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CE-0-0, dag_ret:0, dag_status:2, start_time:1727337625653048, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.656615] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=4] task finish process(ret=0, start_time=1727337625653064, end_time=1727337625656615, runtime=3551, *this={this:0x7f9264d7cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CE-0-0, dag_ret:0, dag_status:2, start_time:1727337625653048, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.656648] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=28][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.656684] ERROR issue_dba_error (ob_log.cpp:1875) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.656687] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.656696] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=0] partition merge iter row count(i=0, row_count=182, ghost_row_count=0, pkey={tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337625620133, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.656727] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625656450, end_time=1727337625656727, runtime=277, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D7-0-0, dag_ret:0, dag_status:2, start_time:1727337625655701, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.656736] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1035, dag_cnt=19, dag_cnts_[dag.get_type()]=19, &dag=0x7f9264d7f6a0, dag={ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D7-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625655701, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.656743] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D7-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625655698, task_id:YB42C0A8002D-0006230056D8C5D7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.656764] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=1] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.656771] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=4] schedule one task(task=0x7f926652a750, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.656776] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625656774, task_id:YB42C0A8002D-0006230056D8C5DA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.656778] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=2] schedule one task(task=0x7f9264d809f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.656813] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=342, range=0x7f92aad52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:21, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:23}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:6, allocator:0x7f9269ffa7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:23, datum_buffer:{capacity:32, datums:0x7f9263a38390, local_datums:0x7f9263a38390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,]}) [2024-09-26 16:00:25.656816] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=333, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a60060, count:7, allocator:0x7f92bc9047d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9264a603b0, local_datums:0x7f9264a603b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:25.656842] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.656875] ERROR issue_dba_error (ob_log.cpp:1875) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.656878] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.656885] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337625619950, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.656901] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625656784, end_time=1727337625656901, runtime=117, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CE-0-0, dag_ret:0, dag_status:2, start_time:1727337625653048, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.656909] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3861, dag_cnt=18, dag_cnts_[dag.get_type()]=18, &dag=0x7f9264d7c7f0, dag={ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5CE-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625653048, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.656916] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5CE-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625653045, task_id:YB42C0A8002D-0006230056D8C5CE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.656970] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.657002] ERROR issue_dba_error (ob_log.cpp:1875) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.657004] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.657013] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=0] partition merge iter row count(i=0, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337625620105, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.657031] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337625656789, end_time=1727337625657031, runtime=242, *this={this:0x7f926652a750, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D6-0-0, dag_ret:0, dag_status:2, start_time:1727337625655693, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.657039] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1346, dag_cnt=17, dag_cnts_[dag.get_type()]=17, &dag=0x7f9264d7f170, dag={ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D6-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625655693, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.657045] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D6-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625655689, task_id:YB42C0A8002D-0006230056D8C5D6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.657062] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625657061, task_id:YB42C0A8002D-0006230056D8C5DB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.657065] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=2] schedule one task(task=0x7f9264d80f20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.657068] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625657068, task_id:YB42C0A8002D-0006230056D8C5DC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.657070] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=2] schedule one task(task=0x7f9264d81450, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.657087] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.657090] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.657092] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:25.657095] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.657098] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337625620210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.657137] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=39] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.657143] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241548806527000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.657311] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752511928, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.657325] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.657329] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.657335] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:121}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752511928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.657352] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:121}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625657083, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=250us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.657375] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DB-0-0, dag_ret:0, dag_status:2, start_time:1727337625657064, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f92684b1480, output_block_cnt_arr:0x7f92684b1488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337625657083, estimated_finish_time:1727337642657375}) [2024-09-26 16:00:25.657382] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d80f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DB-0-0, dag_ret:0, dag_status:2, start_time:1727337625657064, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:25.657386] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=4] task finish process(ret=0, start_time=1727337625657081, end_time=1727337625657386, runtime=305, *this={this:0x7f9264d80f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DB-0-0, dag_ret:0, dag_status:2, start_time:1727337625657064, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:25.657404] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=2] schedule one task(task=0x7f9264d84ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.657474] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.657479] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.657482] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:25.657487] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.657480] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=121, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fe060, count:5, allocator:0x7f92865547d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fe360, local_datums:0x7f92665fe360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:25.657497] INFO [STORAGE] dump_info (ob_sstable_struct.cpp:262) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=13] dump merge info(msg="macro block builder close", output_row_per_s=0, new_macro_KB_per_s=0, *this={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:69, merge_start_time:1727337625657427, merge_finish_time:1727337625657496, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:0, max:0, sum:0, count:0}}, {type:"cost_time", info:{min:0, max:0, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:0, max:0, sum:0, count:0}}, {type:"incremental_row_count", info:{min:0, max:0, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:0, snapshot_version:0, start_scn:0, end_scn:0}, macro_id_list:"", comment:""}) [2024-09-26 16:00:25.657511] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=13] partition merge iter row count(i=0, row_count=0, ghost_row_count=0, pkey={tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, table={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337625620210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.657523] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1544) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=9] merge macro blocks ok(idx_=0, task={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DB-0-0, dag_ret:0, dag_status:2, start_time:1727337625657064, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:25.657491] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b861fe60, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:491, tablet_addr:[816017-1849-0-2305843630912028672-2](offset=1818624,size=579,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:215}, data_tablet_id:{id:215}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3095927781, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86204f4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844118390796288-0](offset=1798144,size=1033,type=2,seq=0), type:"MAJOR", tablet_id:{id:215}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389038459000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843910084883465-0](offset=1799177,size=839,type=2,seq=0), type:"MINOR", tablet_id:{id:215}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726209400113641000, upper_trans_version:1726209400113641000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337625620232, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcf430, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209475373}, this:0x7f92a3bcf430, timestamp:1727334209475373, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:56, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277800266460000, v:0}, rec_scn:{val:1726241800098150000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697789952-2](offset=1810432,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834327372-0](offset=1810764,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881157084-0](offset=1810908,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447677118-0](offset=1811134,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc86d0, 0x7f92a3bcf430, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.657532] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=41] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.657533] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=5] task finish process(ret=0, start_time=1727337625657425, end_time=1727337625657533, runtime=108, *this={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DB-0-0, dag_ret:0, dag_status:2, start_time:1727337625657064, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:25.657539] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.657565] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.657567] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.657569] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:25.657572] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.657574] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.657578] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.657580] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337625620174, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.657597] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=14, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.657607] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.657611] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.657615] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:451}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:16, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.657619] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=101065, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:6, allocator:0x7f92636407d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9263b5a390, local_datums:0x7f9263b5a390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:25.657630] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:451}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:451}, table_count:1, [{i:0, table_key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8650e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625656381, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.23ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.657652] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.657650] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D9-0-0, dag_ret:0, dag_status:2, start_time:1727337625656372, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3470, output_block_cnt_arr:0x7f9287db3478, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337625656381, estimated_finish_time:1727337642657650}) [2024-09-26 16:00:25.657656] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d804c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D9-0-0, dag_ret:0, dag_status:2, start_time:1727337625656372, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.657660] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=4] task finish process(ret=0, start_time=1727337625656380, end_time=1727337625657660, runtime=1280, *this={this:0x7f9264d804c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D9-0-0, dag_ret:0, dag_status:2, start_time:1727337625656372, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.657687] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.657690] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.657710] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=3] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.657715] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=4] schedule one task(task=0x7f9264d850f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.657699] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=0] partition merge iter row count(i=0, row_count=15, ghost_row_count=0, pkey={tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337625620085, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.657742] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625657590, end_time=1727337625657741, runtime=151, *this={this:0x7f926652a3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D5-0-0, dag_ret:0, dag_status:2, start_time:1727337625655358, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.657753] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2394, dag_cnt=16, dag_cnts_[dag.get_type()]=16, &dag=0x7f9264d7ec40, dag={ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D5-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625655358, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.657749] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=451, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:14, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:16}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685ea060, count:4, allocator:0x7f92bc9047d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:16, datum_buffer:{capacity:32, datums:0x7f92685ea350, local_datums:0x7f92685ea350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,]}) [2024-09-26 16:00:25.657760] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D5-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625655354, task_id:YB42C0A8002D-0006230056D8C5D5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.657791] INFO [STORAGE.COMPACTION] get_merged_sstable (ob_tablet_merge_task.cpp:1285) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=5] create new merged sstable(ctx.param_.tablet_id_={id:121}, snapshot_version=1726241132764173999, ctx.param_.merge_type_=3, ctx.create_snapshot_version_=0, table_mode_flag=0) [2024-09-26 16:00:25.657798] INFO [STORAGE] close (ob_index_block_builder.cpp:815) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=4] succeed to close sstable index builder(res={root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_block_ids_.count():0, other_block_ids_.count():0, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, data_column_cnt:10, data_column_checksums:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], row_count:0, max_merged_trans_version:0, contain_uncommitted_row:false, occupy_size:0, original_size:0, data_checksum:0, use_old_macro_block_count:0, compressor_type:1, root_row_store_type:0, nested_offset:0, nested_size:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}) [2024-09-26 16:00:25.657812] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.657815] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.657812] INFO [STORAGE] init (ob_sstable.cpp:118) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=8] succeeded to init sstable(ret=0, this={ObITable:{this:0x7f92684b0128, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f92684b0128, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:null, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:25.657824] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337625620174, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.657830] INFO [STORAGE.COMPACTION] create_sstable (ob_tablet_merge_ctx.cpp:367) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=17] succeed to merge sstable(param={table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_logic_seq:0, schema_version:1710828752511928, create_snapshot_version:0, progressive_merge_round:0, progressive_merge_step:0, is_ready_for_read:true, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, index_type:0, root_block_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), root_block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, root_row_store_type:0, latest_row_store_type:1, data_index_tree_height:0, data_block_macro_meta_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), data_block_macro_meta:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, use_old_macro_block_count:0, row_count:0, rowkey_column_cnt:5, column_cnt:10, column_checksums:[], data_checksum:0, occupy_size:0, original_size:0, max_merged_trans_version:1726241132764173999, ddl_scn:{val:0, v:0}, contain_uncommitted_row:false, is_meta_root:false, compressor_type:1, encrypt_id:0, master_key_id:0, recycle_version:0, nested_offset:0, nested_size:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}, table_key={tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_merge_info={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:413, merge_start_time:1727337625657083, merge_finish_time:1727337625657496, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"cost_time", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"incremental_row_count", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:1, snapshot_version:0, start_scn:1726211867305630000, end_scn:1726241548806527000}, macro_id_list:"", comment:""}) [2024-09-26 16:00:25.657875] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625657725, end_time=1727337625657868, runtime=143, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D9-0-0, dag_ret:0, dag_status:2, start_time:1727337625656372, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.657881] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337625620232, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.657886] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1513, dag_cnt=15, dag_cnts_[dag.get_type()]=15, &dag=0x7f9264d80100, dag={ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D9-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625656372, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.657893] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D9-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625656367, task_id:YB42C0A8002D-0006230056D8C5D9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.657890] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752725472, storage_schema:{this:0x7f92665fd330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752725472, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.657901] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.657904] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.657909] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92665fd330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:215}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752725472, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.657926] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625657924, task_id:YB42C0A8002D-0006230056D8C5DD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.657929] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=3] schedule one task(task=0x7f9264d81980, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.657933] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625657932, task_id:YB42C0A8002D-0006230056D8C5DE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.657935] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=2] schedule one task(task=0x7f9264d81eb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.657924] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:215}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752725472, storage_schema:{this:0x7f92665fd330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:215}, table_count:1, [{i:0, table_key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625657451, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=456us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.657944] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DC-0-0, dag_ret:0, dag_status:2, start_time:1727337625657070, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665fd460, output_block_cnt_arr:0x7f92665fd468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337625657451, estimated_finish_time:1727337642657944}) [2024-09-26 16:00:25.657951] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.657950] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d81450, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DC-0-0, dag_ret:0, dag_status:2, start_time:1727337625657070, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.657954] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.657956] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:25.657955] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=5] task finish process(ret=0, start_time=1727337625657448, end_time=1727337625657955, runtime=507, *this={this:0x7f9264d81450, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DC-0-0, dag_ret:0, dag_status:2, start_time:1727337625657070, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.657960] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.657959] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.657963] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.657965] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:25.657969] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.657971] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.657972] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.657972] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337625620154, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.657978] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.657979] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.657979] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=19, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.658014] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.658017] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.658021] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:334}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:21, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.658045] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=2] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.658037] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:334}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:334}, table_count:1, [{i:0, table_key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8654c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625657561, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=457us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.658081] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=215, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:6}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:3, allocator:0x7f92691a27d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92664f0320, local_datums:0x7f92664f0320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:25.658094] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=24] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D8-0-0, dag_ret:0, dag_status:2, start_time:1727337625655919, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a434a0, output_block_cnt_arr:0x7f9263a434a8, concurrent_cnt:1, estimate_row_cnt:993, estimate_occupy_size:0, latest_update_ts:1727337625657561, estimated_finish_time:1727337642658094}) [2024-09-26 16:00:25.658100] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7ff90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D8-0-0, dag_ret:0, dag_status:2, start_time:1727337625655919, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.658104] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=4] task finish process(ret=0, start_time=1727337625657560, end_time=1727337625658104, runtime=544, *this={this:0x7f9264d7ff90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D8-0-0, dag_ret:0, dag_status:2, start_time:1727337625655919, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.658137] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.658152] INFO [STORAGE] build_new_table_store (ob_tablet_table_store.cpp:1206) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=18] succeed to build new table store(major_tables_={cnt:1, is_inited:true}, minor_tables_={cnt:3, is_inited:true}, memtables_={count:1}, PRINT_TS(*this)={ObTabletTableStore_Pretty:this:0x7f92aac55ca8, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, memtables:{count:1}, is_ready_for_read:true, table_array:{ [2856] [ YB42C0A8002D-0006230056D8C5DB-0-0 ] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [2856] [ YB42C0A8002D-0006230056D8C5DB-0-0 ] MAJOR MAJOR 1725120003420849000 1725120003420849000 {val:0, v:0} {val:1726077600415496000, v:0} 0 false [2856] [ YB42C0A8002D-0006230056D8C5DB-0-0 ] MINOR MINOR 1726178585269925002 1726178585269925002 {val:1726077259159067000, v:0} {val:1726178585269925002, v:0} 0 false [2856] [ YB42C0A8002D-0006230056D8C5DB-0-0 ] MINI 1726211867106157000 1726211867106157000 {val:1726178585269925002, v:0} {val:1726211867305630000, v:0} 0 false [2856] [ YB42C0A8002D-0006230056D8C5DB-0-0 ] MINI 1726241132764173999 1726241132764173999 {val:1726211867305630000, v:0} {val:1726241548806527000, v:0} 0 false [2856] [ YB42C0A8002D-0006230056D8C5DB-0-0 ] MEM ACTIVE 9223372036854775807 18446744073709547609 {val:1726241548806527000, v:0} {val:4611686018427387903, v:0} 3 unused }}) [2024-09-26 16:00:25.658169] ERROR issue_dba_error (ob_log.cpp:1875) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.658171] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.658189] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=0] partition merge iter row count(i=0, row_count=9, ghost_row_count=0, pkey={tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337625620232, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.658201] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.658212] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=11][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.658215] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:25.658220] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.658224] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.658232] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.658257] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625658056, end_time=1727337625658257, runtime=201, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DC-0-0, dag_ret:0, dag_status:2, start_time:1727337625657070, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.658268] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1198, dag_cnt=14, dag_cnts_[dag.get_type()]=14, &dag=0x7f9264d81090, dag={ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DC-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625657070, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.658275] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DC-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625657068, task_id:YB42C0A8002D-0006230056D8C5DC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.658331] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337625620253, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.658340] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753355744, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753355744, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.658353] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.658355] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.658360] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:456}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753355744, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.658377] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:456}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753355744, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:456}, table_count:1, [{i:0, table_key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8634c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625657947, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=411us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.658402] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DD-0-0, dag_ret:0, dag_status:2, start_time:1727337625657929, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a45490, output_block_cnt_arr:0x7f9263a45498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337625657947, estimated_finish_time:1727337642658402}) [2024-09-26 16:00:25.658410] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d81980, type:15, status:2, dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DD-0-0, dag_ret:0, dag_status:2, start_time:1727337625657929, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.658407] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337625620269, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.658415] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=5] task finish process(ret=0, start_time=1727337625657945, end_time=1727337625658415, runtime=470, *this={this:0x7f9264d81980, type:15, status:2, dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DD-0-0, dag_ret:0, dag_status:2, start_time:1727337625657929, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.658417] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.658428] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=3] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.658429] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.658433] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=2] schedule one task(task=0x7f9264d85260, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.658438] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625658436, task_id:YB42C0A8002D-0006230056D8C5DF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.658440] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=2] schedule one task(task=0x7f9264d823e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.658497] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.658500] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.658502] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:25.658491] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=334, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:19, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:21}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:8, allocator:0x7f92636767d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:21, datum_buffer:{capacity:32, datums:0x7f9263b5a3e0, local_datums:0x7f9263b5a3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,]}) [2024-09-26 16:00:25.658505] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.658508] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.658514] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.658569] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=456, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fa060, count:7, allocator:0x7f926367e7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f92665fa3b0, local_datums:0x7f92665fa3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:25.658629] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.658663] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.658665] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.658673] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=0] partition merge iter row count(i=0, row_count=45, ghost_row_count=0, pkey={tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337625620253, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.658689] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.658689] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625658544, end_time=1727337625658689, runtime=145, *this={this:0x7f9264d85260, type:1, status:2, dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DD-0-0, dag_ret:0, dag_status:2, start_time:1727337625657929, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.658698] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=769, dag_cnt=13, dag_cnts_[dag.get_type()]=13, &dag=0x7f9264d815c0, dag={ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DD-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625657929, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.658705] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DD-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625657924, task_id:YB42C0A8002D-0006230056D8C5DD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.658724] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.658727] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.658744] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625658742, task_id:YB42C0A8002D-0006230056D8C5E0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.658736] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=0] partition merge iter row count(i=0, row_count=153, ghost_row_count=0, pkey={tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337625620154, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.658747] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=3] schedule one task(task=0x7f9264d82910, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.658770] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625658464, end_time=1727337625658769, runtime=305, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D8-0-0, dag_ret:0, dag_status:2, start_time:1727337625655919, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.658780] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2861, dag_cnt=12, dag_cnts_[dag.get_type()]=12, &dag=0x7f9264d7fbd0, dag={ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5D8-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625655919, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.658789] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5D8-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337625655914, task_id:YB42C0A8002D-0006230056D8C5D8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.658802] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337625658801, task_id:YB42C0A8002D-0006230056D8C5E1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.658804] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=2] schedule one task(task=0x7f9264d82e40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.658817] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.658819] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.658821] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:25.658825] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.658828] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b863eec0, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:499, tablet_addr:[816017-1849-0-2305843639502065664-2](offset=1921024,size=587,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101016}, data_tablet_id:{id:154}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726074677203981000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:398898191, row_count:578}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844419038609408-0](offset=1900544,size=1313,type=2,seq=0), type:"MAJOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726074015354550000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844024975361313-0](offset=1901857,size=946,type=2,seq=0), type:"MINOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:1726074677203981000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726205802467242000, upper_trans_version:1726205802467242000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337625620308, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd5fc0, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209508808}, this:0x7f92a3bd5fc0, timestamp:1727334209508808, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:69, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277803107894000, v:0}, rec_scn:{val:1726241804194470000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771625984-2](offset=1904640,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161774413-0](offset=1904973,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881251311-0](offset=1905135,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc7f00, 0x7f92a3bd5fc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.658876] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=48] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.658882] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.658906] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.658913] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:154}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.658923] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337625620288, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.658933] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.658945] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.658933] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:154}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:154}, table_count:1, [{i:0, table_key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8623ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625657955, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=956us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.658948] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.658958] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DE-0-0, dag_ret:0, dag_status:2, start_time:1727337625657935, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515480, output_block_cnt_arr:0x7f9268515488, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337625657955, estimated_finish_time:1727337642658958}) [2024-09-26 16:00:25.658953] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101015}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.658963] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d81eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DE-0-0, dag_ret:0, dag_status:2, start_time:1727337625657935, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.658968] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=5] task finish process(ret=0, start_time=1727337625657953, end_time=1727337625658968, runtime=1015, *this={this:0x7f9264d81eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DE-0-0, dag_ret:0, dag_status:2, start_time:1727337625657935, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.658970] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:101015}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101015}, table_count:1, [{i:0, table_key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625658494, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=457us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.659026] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=25] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DF-0-0, dag_ret:0, dag_status:2, start_time:1727337625658440, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab490, output_block_cnt_arr:0x7f92684ab498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337625658494, estimated_finish_time:1727337642659026}) [2024-09-26 16:00:25.659034] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d823e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DF-0-0, dag_ret:0, dag_status:2, start_time:1727337625658440, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.659038] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=4] task finish process(ret=0, start_time=1727337625658492, end_time=1727337625659038, runtime=546, *this={this:0x7f9264d823e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DF-0-0, dag_ret:0, dag_status:2, start_time:1727337625658440, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.659053] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=2] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.659057] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=3] schedule one task(task=0x7f9262cae030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.659093] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=154, range=0x7f92aaad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:5, allocator:0x7f92a31f67d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:25.659104] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101015, range=0x7f92a9cd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c70060, count:6, allocator:0x7f92bc9047d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9262c70390, local_datums:0x7f9262c70390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:25.659196] ERROR issue_dba_error (ob_log.cpp:1875) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.659198] ERROR issue_dba_error (ob_log.cpp:1875) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.659198] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.659201] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.659207] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337625620288, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.659210] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337625620269, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.659226] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625659076, end_time=1727337625659225, runtime=149, *this={this:0x7f9262cae030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DF-0-0, dag_ret:0, dag_status:2, start_time:1727337625658440, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.659228] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625659067, end_time=1727337625659228, runtime=161, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DE-0-0, dag_ret:0, dag_status:2, start_time:1727337625657935, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.659234] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=794, dag_cnt=11, dag_cnts_[dag.get_type()]=11, &dag=0x7f9264d82020, dag={ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DF-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625658440, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.659241] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DF-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625658436, task_id:YB42C0A8002D-0006230056D8C5DF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.659265] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1330, dag_cnt=10, dag_cnts_[dag.get_type()]=10, &dag=0x7f9264d81af0, dag={ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DE-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625657935, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.659274] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DE-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337625657932, task_id:YB42C0A8002D-0006230056D8C5DE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.659291] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337625659290, task_id:YB42C0A8002D-0006230056D8C5E2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.659294] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=2] schedule one task(task=0x7f9264d83370, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.659300] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625659299, task_id:YB42C0A8002D-0006230056D8C5E3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.659302] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=2] schedule one task(task=0x7f9264d838a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.659318] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.659321] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.659323] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:25.659326] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.659330] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.659336] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.659526] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337625620362, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.659536] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=10] update column cnt(column_cnt_=26, store_column_cnt_=26, column_cnt_=26, column_array_.count()=0) [2024-09-26 16:00:25.659539] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92684ab480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=26, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.659550] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.659553] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.659558] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:328}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:28, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.659574] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:328}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92684ab480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:328}, table_count:1, [{i:0, table_key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e072bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625659314, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=241us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.659595] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d834e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E3-0-0, dag_ret:0, dag_status:2, start_time:1727337625659302, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab5d0, output_block_cnt_arr:0x7f92684ab5d8, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337625659314, estimated_finish_time:1727337642659594}) [2024-09-26 16:00:25.659600] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d838a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d834e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E3-0-0, dag_ret:0, dag_status:2, start_time:1727337625659302, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.659605] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=5] task finish process(ret=0, start_time=1727337625659313, end_time=1727337625659605, runtime=292, *this={this:0x7f9264d838a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d834e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E3-0-0, dag_ret:0, dag_status:2, start_time:1727337625659302, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.659633] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=2] schedule one task(task=0x7f9264d85260, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.659677] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=328, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:26, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:28}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ee060, count:5, allocator:0x7f92c19447d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:28, datum_buffer:{capacity:32, datums:0x7f92664ee360, local_datums:0x7f92664ee360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,]}) [2024-09-26 16:00:25.659708] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.659738] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.659740] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.659748] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337625620362, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.659763] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625659652, end_time=1727337625659763, runtime=111, *this={this:0x7f9264d85260, type:1, status:2, dag:{ObIDag:{this:0x7f9264d834e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E3-0-0, dag_ret:0, dag_status:2, start_time:1727337625659302, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.659771] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=469, dag_cnt=9, dag_cnts_[dag.get_type()]=9, &dag=0x7f9264d834e0, dag={ObIDag:{this:0x7f9264d834e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E3-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625659302, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.659778] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E3-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625659299, task_id:YB42C0A8002D-0006230056D8C5E3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.659789] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625659787, task_id:YB42C0A8002D-0006230056D8C5E4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.659792] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=3] schedule one task(task=0x7f9264d83dd0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.659805] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.659808] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.659810] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:25.659814] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.659816] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.659822] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.660312] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337625620383, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.660328] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337625620308, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.660344] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=30] update column cnt(column_cnt_=12, store_column_cnt_=12, column_cnt_=12, column_array_.count()=0) [2024-09-26 16:00:25.660344] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.660351] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.660348] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9287db3490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.660356] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.660359] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:25.660360] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.660363] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.660365] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.660370] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.660374] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.660370] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101016}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.660380] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:330}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.660389] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101016}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101016}, table_count:1, [{i:0, table_key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625658812, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.56ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.660367] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b864fe60, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:510, tablet_addr:[816017-1849-0-2305843651313397760-2](offset=2093056,size=598,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101063}, data_tablet_id:{id:297}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726063409221604000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1864622635, row_count:24}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86505a8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844568288894976-0](offset=2072576,size=1452,type=2,seq=0), type:"MAJOR", tablet_id:{id:101063}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063202518238000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843928338769324-0](offset=2074028,size=856,type=2,seq=0), type:"MINI", tablet_id:{id:101063}, scn_range:{start_scn:{val:1726063409221604000, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557514739000, upper_trans_version:1726121557514739000}, {i:1, addr:[816017-1849-0-2305843928338770180-0](offset=2074884,size=856,type=2,seq=0), type:"MINI", tablet_id:{id:101063}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149603136610000, upper_trans_version:1726149603136610000}, {i:2, addr:[816017-1849-0-2305843927265029212-0](offset=2075740,size=855,type=2,seq=0), type:"MINI", tablet_id:{id:101063}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726207962418314000, upper_trans_version:1726207962418314000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337625620325, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd8ea0, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220519573}, this:0x7f92a3bd8ea0, timestamp:1727334220519573, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:75, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726264804632655000, v:0}, rec_scn:{val:1726264804632655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843593331331072-2](offset=2084864,size=544,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046856736-0](offset=2085408,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881431758-0](offset=2085582,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bca610, 0x7f92a3bd8ea0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.660411] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=44] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.660400] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:330}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9287db3490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:330}, table_count:1, [{i:0, table_key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e107fe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625659802, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=576us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.660417] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.660416] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E0-0-0, dag_ret:0, dag_status:2, start_time:1727337625658747, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1490, output_block_cnt_arr:0x7f92728f1498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337625658812, estimated_finish_time:1727337642660416}) [2024-09-26 16:00:25.660423] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d82910, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E0-0-0, dag_ret:0, dag_status:2, start_time:1727337625658747, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.660429] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=5] task finish process(ret=0, start_time=1727337625658811, end_time=1727337625660428, runtime=1617, *this={this:0x7f9264d82910, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E0-0-0, dag_ret:0, dag_status:2, start_time:1727337625658747, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.660436] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.660438] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.660440] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:25.660443] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.660446] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E4-0-0, dag_ret:0, dag_status:2, start_time:1727337625659792, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db35f0, output_block_cnt_arr:0x7f9287db35f8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337625659802, estimated_finish_time:1727337642660438}) [2024-09-26 16:00:25.660454] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d83dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E4-0-0, dag_ret:0, dag_status:2, start_time:1727337625659792, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.660474] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=20] task finish process(ret=0, start_time=1727337625659800, end_time=1727337625660474, runtime=674, *this={this:0x7f9264d83dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E4-0-0, dag_ret:0, dag_status:2, start_time:1727337625659792, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.660445] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8652d40, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[816017-1849-0-2305843644870942720-2](offset=2088960,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:452}, data_tablet_id:{id:452}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2139885871, row_count:290}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b865341c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844508159356928-0](offset=2076672,size=1396,type=2,seq=0), type:"MAJOR", tablet_id:{id:452}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562830808000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843992763282804-0](offset=2078068,size=916,type=2,seq=0), type:"MINI", tablet_id:{id:452}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986149768000, upper_trans_version:1726149986149768000}, {i:1, addr:[816017-1849-0-2305844002426960136-0](offset=2078984,size=925,type=2,seq=0), type:"MINI", tablet_id:{id:452}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726208018712107000, upper_trans_version:1726208018712107000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337625620344, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd9670, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220578716}, this:0x7f92a3bd9670, timestamp:1727334220578716, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:76, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265292959510000, v:0}, rec_scn:{val:1726264902486897000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843480588435456-2](offset=2080768,size=439,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719499703-0](offset=2081207,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881427539-0](offset=2081363,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447947573-0](offset=2081589,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc9e40, 0x7f92a3bd9670, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.660502] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.660507] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.660523] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337625620192, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.660534] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=2] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.660530] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.660541] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=4] schedule one task(task=0x7f9264d85260, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.660543] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.660546] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.660551] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:378}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.660569] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:378}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:378}, table_count:1, [{i:0, table_key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625658197, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.35ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.660591] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DA-0-0, dag_ret:0, dag_status:2, start_time:1727337625656778, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db9480, output_block_cnt_arr:0x7f9287db9488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337625658197, estimated_finish_time:1727337642660591}) [2024-09-26 16:00:25.660587] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=13] multi version scan iterator init succ(param.table_id_=330, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d30060, count:7, allocator:0x7f92864067d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9287d303b0, local_datums:0x7f9287d303b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:25.660598] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d809f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DA-0-0, dag_ret:0, dag_status:2, start_time:1727337625656778, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.660603] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=5] task finish process(ret=0, start_time=1727337625658196, end_time=1727337625660603, runtime=2407, *this={this:0x7f9264d809f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DA-0-0, dag_ret:0, dag_status:2, start_time:1727337625656778, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.660612] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=2] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.660652] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=378, range=0x7f92ab0d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:5, allocator:0x7f92c4a5e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:25.660713] ERROR issue_dba_error (ob_log.cpp:1875) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.660716] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.660750] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.660724] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337625620192, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.660784] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625660622, end_time=1727337625660783, runtime=161, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DA-0-0, dag_ret:0, dag_status:2, start_time:1727337625656778, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.660804] ERROR issue_dba_error (ob_log.cpp:1875) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=8][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.660806] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4028, dag_cnt=8, dag_cnts_[dag.get_type()]=8, &dag=0x7f9264d80630, dag={ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DA-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625656778, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.660808] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.660815] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DA-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337625656774, task_id:YB42C0A8002D-0006230056D8C5DA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.660819] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=0] partition merge iter row count(i=0, row_count=141, ghost_row_count=0, pkey={tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337625620383, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.660841] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625660555, end_time=1727337625660841, runtime=286, *this={this:0x7f9264d85260, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E4-0-0, dag_ret:0, dag_status:2, start_time:1727337625659792, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.660874] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625660872, task_id:YB42C0A8002D-0006230056D8C5E5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.660878] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=3] schedule one task(task=0x7f9264d84300, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.660908] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101016, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ee060, count:6, allocator:0x7f92636447d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92664ee390, local_datums:0x7f92664ee390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:25.660994] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.660997] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.661005] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337625620308, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.661023] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625660867, end_time=1727337625661023, runtime=156, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E0-0-0, dag_ret:0, dag_status:2, start_time:1727337625658747, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.661032] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2285, dag_cnt=7, dag_cnts_[dag.get_type()]=7, &dag=0x7f9264d82550, dag={ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E0-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625658747, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.661040] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E0-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625658742, task_id:YB42C0A8002D-0006230056D8C5E0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.661055] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625661054, task_id:YB42C0A8002D-0006230056D8C5E6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.661058] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=3] schedule one task(task=0x7f9264d84830, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.661090] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.661093] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.661094] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.661096] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:25.661098] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.661100] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:25.661101] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.661104] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.661104] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8648b00, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[816017-1849-0-2305843650239631360-2](offset=2068480,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:332}, data_tablet_id:{id:332}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:235179760, row_count:171}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8649200, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844652040736768-0](offset=2052096,size=1530,type=2,seq=0), type:"MAJOR", tablet_id:{id:332}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843956256036346-0](offset=2053626,size=882,type=2,seq=0), type:"MINI", tablet_id:{id:332}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}, {i:1, addr:[816017-1849-0-2305843965919713644-0](offset=2054508,size=891,type=2,seq=0), type:"MINI", tablet_id:{id:332}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726208018709503000, upper_trans_version:1726208018709503000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337625620435, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdbd80, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220854400}, this:0x7f92a3bdbd80, timestamp:1727334220854400, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:81, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265274283037001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479514673152-2](offset=2060288,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161930166-0](offset=2060726,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881407064-0](offset=2060888,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447927098-0](offset=2061114,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bcade0, 0x7f92a3bdbd80, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.661108] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706339704000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f926e089f20, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:510, tablet_addr:[809012-1445-0-2305843651311779840-2](offset=475136,size=598,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:329}, data_tablet_id:{id:329}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706339704000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:305104969, row_count:10474}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f926e08a620, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844778740674560-0](offset=454656,size=1648,type=2,seq=0), type:"MAJOR", tablet_id:{id:329}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562805980000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843951959471728-0](offset=456304,size=878,type=2,seq=0), type:"MINI", tablet_id:{id:329}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986144852000, upper_trans_version:1726149986144852000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337625620399, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdec60, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221352878}, this:0x7f92a3bdec60, timestamp:1727334221352878, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:87, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265292878107000, v:0}, rec_scn:{val:1726264907253071001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843366770180096-2](offset=458752,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843183160328525-0](offset=459085,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879805423-0](offset=459247,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446325457-0](offset=459473,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bcb5b0, 0x7f92a3bdec60, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.661157] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=49] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.661157] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=53] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.661164] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.661167] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.661180] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1388, dag_cnt=6, dag_cnts_[dag.get_type()]=6, &dag=0x7f9264d83a10, dag={ObIDag:{this:0x7f9264d83a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E4-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625659792, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.661189] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E4-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337625659787, task_id:YB42C0A8002D-0006230056D8C5E4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.661232] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337625661230, task_id:YB42C0A8002D-0006230056D8C5E7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.661236] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=4] schedule one task(task=0x7f9264d84d60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.661379] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337625620435, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.661388] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=25, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.661416] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=24] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.661420] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.661426] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:332}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:27, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.661444] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:332}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:332}, table_count:1, [{i:0, table_key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8648b00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625661084, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=339us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.661466] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d84470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E6-0-0, dag_ret:0, dag_status:2, start_time:1727337625661058, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1490, output_block_cnt_arr:0x7f92728f1498, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337625661084, estimated_finish_time:1727337642661466}) [2024-09-26 16:00:25.661472] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d84830, type:15, status:2, dag:{ObIDag:{this:0x7f9264d84470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E6-0-0, dag_ret:0, dag_status:2, start_time:1727337625661058, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.661477] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=5] task finish process(ret=0, start_time=1727337625661082, end_time=1727337625661477, runtime=395, *this={this:0x7f9264d84830, type:15, status:2, dag:{ObIDag:{this:0x7f9264d84470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E6-0-0, dag_ret:0, dag_status:2, start_time:1727337625661058, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.661540] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=2] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.661540] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337625620399, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.661551] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=10] update column cnt(column_cnt_=29, store_column_cnt_=29, column_cnt_=29, column_array_.count()=0) [2024-09-26 16:00:25.661554] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9264a69490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.661566] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.661569] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.661575] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264a69490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:329}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.661578] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=332, range=0x7f92aab52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:25, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:27}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:6, allocator:0x7f92c19247d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:27, datum_buffer:{capacity:32, datums:0x7f9264a62390, local_datums:0x7f9264a62390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,]}) [2024-09-26 16:00:25.661593] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:329}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9264a69490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:329}, table_count:1, [{i:0, table_key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e089f20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625661088, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=484us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.661612] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.661618] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E5-0-0, dag_ret:0, dag_status:2, start_time:1727337625660877, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264a695f0, output_block_cnt_arr:0x7f9264a695f8, concurrent_cnt:1, estimate_row_cnt:62, estimate_occupy_size:0, latest_update_ts:1727337625661088, estimated_finish_time:1727337642661617}) [2024-09-26 16:00:25.661624] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d84300, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E5-0-0, dag_ret:0, dag_status:2, start_time:1727337625660877, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.661628] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=4] task finish process(ret=0, start_time=1727337625661087, end_time=1727337625661628, runtime=541, *this={this:0x7f9264d84300, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E5-0-0, dag_ret:0, dag_status:2, start_time:1727337625660877, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.661660] ERROR issue_dba_error (ob_log.cpp:1875) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.661664] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.661675] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337625620435, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.661695] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625661551, end_time=1727337625661695, runtime=144, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d84470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E6-0-0, dag_ret:0, dag_status:2, start_time:1727337625661058, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.661703] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=645, dag_cnt=5, dag_cnts_[dag.get_type()]=5, &dag=0x7f9264d84470, dag={ObIDag:{this:0x7f9264d84470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E6-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625661058, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.661712] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E6-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337625661054, task_id:YB42C0A8002D-0006230056D8C5E6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.661729] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=3] schedule one task(task=0x7f9264c3a3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.661822] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337625620344, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.661835] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f92728f7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=15, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.661848] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.661851] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.661857] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:452}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:17, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.661874] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:452}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f92728f7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:452}, table_count:1, [{i:0, table_key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8652d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625660433, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.42ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.661897] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E2-0-0, dag_ret:0, dag_status:2, start_time:1727337625659293, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f7480, output_block_cnt_arr:0x7f92728f7488, concurrent_cnt:1, estimate_row_cnt:10, estimate_occupy_size:0, latest_update_ts:1727337625660433, estimated_finish_time:1727337642661897}) [2024-09-26 16:00:25.661903] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d83370, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E2-0-0, dag_ret:0, dag_status:2, start_time:1727337625659293, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.661908] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=5] task finish process(ret=0, start_time=1727337625660432, end_time=1727337625661908, runtime=1476, *this={this:0x7f9264d83370, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E2-0-0, dag_ret:0, dag_status:2, start_time:1727337625659293, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.661917] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=3] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.661937] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:25.661941] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.661953] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=452, range=0x7f92a98d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:15, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:17}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:5, allocator:0x7f92bc90a7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:17, datum_buffer:{capacity:32, datums:0x7f9264a62360, local_datums:0x7f9264a62360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,]}) [2024-09-26 16:00:25.661970] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.661973] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:25.661975] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:25.661979] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:25.662015] ERROR issue_dba_error (ob_log.cpp:1875) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.661982] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b86a2d40, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:2, version:2, length:511, tablet_addr:[816017-1849-0-2305843652387110912-2](offset=2064384,size=599,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101067}, data_tablet_id:{id:334}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1854554341, row_count:13383}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86a3488, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844595132416000-0](offset=2048000,size=1477,type=2,seq=0), type:"MAJOR", tablet_id:{id:101067}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843951961064901-0](offset=2049477,size=878,type=2,seq=0), type:"MINI", tablet_id:{id:101067}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}, {i:1, addr:[816017-1849-0-2305843958403516723-0](offset=2050355,size=884,type=2,seq=0), type:"MINI", tablet_id:{id:101067}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726208018709503000, upper_trans_version:1726208018709503000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337625620468, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bde490, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221276956}, this:0x7f92a3bde490, timestamp:1727334221276956, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:86, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265292880560000, v:0}, rec_scn:{val:1726264918599518000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843482735894528-2](offset=2056192,size=441,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046827961-0](offset=2056633,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881402983-0](offset=2056807,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bcbd80, 0x7f92a3bde490, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:25.662019] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=37] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:25.662018] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.662025] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.662025] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=0] partition merge iter row count(i=0, row_count=10, ghost_row_count=0, pkey={tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337625620344, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.662055] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625661928, end_time=1727337625662055, runtime=127, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E2-0-0, dag_ret:0, dag_status:2, start_time:1727337625659293, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.662064] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2770, dag_cnt=4, dag_cnts_[dag.get_type()]=4, &dag=0x7f9264d82fb0, dag={ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E2-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625659293, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.662070] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E2-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625659290, task_id:YB42C0A8002D-0006230056D8C5E2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.662149] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=329, range=0x7f92aae56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:6, allocator:0x7f92636767d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9263a3a390, local_datums:0x7f9263a3a390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:25.662278] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.662317] ERROR issue_dba_error (ob_log.cpp:1875) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.662320] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.662330] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=0] partition merge iter row count(i=0, row_count=47, ghost_row_count=0, pkey={tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337625620399, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.662353] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625662106, end_time=1727337625662353, runtime=247, *this={this:0x7f9264c3a3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E5-0-0, dag_ret:0, dag_status:2, start_time:1727337625660877, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.662364] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1487, dag_cnt=3, dag_cnts_[dag.get_type()]=3, &dag=0x7f9264d83f40, dag={ObIDag:{this:0x7f9264d83f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E5-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625660877, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.662372] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E5-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625660872, task_id:YB42C0A8002D-0006230056D8C5E5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.662398] INFO [MDS] init_for_evict_medium_info (ob_tablet_mds_data.cpp:414) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=14] succeeded to init mds data(ret=0, finish_medium_scn=1726077600415496000, src_medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, src_extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}) [2024-09-26 16:00:25.662415] INFO [STORAGE] init_for_merge (ob_tablet.cpp:363) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=5] succeeded to init tablet for mini/minor/major merge(ret=0, param={sstable:0x7f92684b0128, snapshot_version:1726241132764173999, clog_checkpoint_scn:{val:1726241548806527000, v:0}, multi_version_start:1726178585269925002, need_report:false, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, rebuild_seq:0, update_with_major_flag:false, need_check_sstable:false, ddl_info:{keep_old_ddl_sstable:true, ddl_start_scn:{val:0, v:0}, ddl_commit_scn:{val:0, v:0}, ddl_checkpoint_scn:{val:0, v:0}, ddl_snapshot_version:0, ddl_execution_id:-1, data_format_version:0}, allow_duplicate_sstable:false, merge_type:"MINI_MERGE", need_check_transfer_seq:true, transfer_seq:0}, old_tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337625619843000, hold_ref_cnt:true, ref_cnt:3, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337625620210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}, this={this:0x7f9287db6060, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x7f92aac56450, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726241548806527000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726241132764173999, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f9287db6670, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}, {i:2, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726241132764173999, upper_trans_version:1726241132764173999}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x7f9287db6520, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9287db7880, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9287db7950, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{redefined:false, snapshot_version:0, schema_version:0, data_tablet_id:{id:0}, hidden_tablet_id:{id:0}, lob_meta_tablet_id:{id:50121}, lob_piece_tablet_id:{id:60121}, is_old_mds:true}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:25.662513] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=82] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287db8060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}}, ref_cnt:0, upper_trans_version:1725120003420849000, timestamp:0}, this:0x7f9287db8060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1725120003420849000, max_merged_trans_version:1725120003420849000, data_macro_block_count:1, nested_size:4096, nested_offset:589824, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:84, row_count_:14, occupy_size_:1604, original_size_:1914, data_checksum_:3587602083, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:1, data_micro_block_count_:1, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:2, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1725120003420849000, max_merged_trans_version_:1725120003420849000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287db8310, column_checksum_count:10, data_root_info:{addr:[0-0-0-3458764933653594112-0](offset=0,size=391,type=3,seq=0), block_data:{buf:0x7f9287db8360, size:391, extra_buf:0x7f9287db84e7, extra_size:424, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-3458765009889263616-0](offset=0,size=462,type=3,seq=0), block_data:{buf:0x7f9287db868f, size:462, extra_buf:null, extra_size:0, type:1}}, data_block_count:1, other_block_count:0, linked_block_count:0, data_block_ids:0x7f9287db885d, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:true, nested_offset:589824, nested_size:4096}}, valid_for_reading:true}) [2024-09-26 16:00:25.662532] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=17] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287db8060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}}, ref_cnt:0, upper_trans_version:1726178585269925002, timestamp:0}, this:0x7f9287db8060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726178585269925002, max_merged_trans_version:1726178585269925002, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726178585269925002, v:0}, meta:{basic_meta:{version:1, length:94, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726178585269925002, max_merged_trans_version_:1726178585269925002, recycle_version:1726077600415496000, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726178585269925002, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:1, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287db8310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:25.662545] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=11] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287db8060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}}, ref_cnt:0, upper_trans_version:1726211867106157000, timestamp:0}, this:0x7f9287db8060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726211867106157000, max_merged_trans_version:1726211867106157000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726211867305630000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726211867106157000, max_merged_trans_version_:1726211867106157000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726211867305630000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287db8310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:25.662554] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=9] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287db8060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9287db8060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287db8310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:25.662564] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=9][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:25.662594] ERROR issue_dba_error (ob_log.cpp:1875) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.662596] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf9de5b1 0xf9dcfd6 0xe51c6fb 0xe519c03 0xe518f89 0xe516f8c 0xf01158d 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.662614] INFO [STORAGE] reset (ob_tablet.cpp:159) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=0] reset tablet(this=0x7f9287db6060, ls_id={id:1}, tablet_id={id:121}, lbt()="0x121f1f9c 0xe52cc09 0xe52c6c5 0xe52bd13 0x4fd348f 0xf01110c 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:25.662621] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1250) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=0][errcode=-4184] sstable merge finish(ret=-4184, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DB-0-0, dag_ret:0, dag_status:2, start_time:1727337625657064, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f92684b1480, output_block_cnt_arr:0x7f92684b1488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337625657083, estimated_finish_time:1727337642657375}, compaction_filter:NULL, time_guard:total=766us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, task={this:0x7f9264d850f0, type:16, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DB-0-0, dag_ret:0, dag_status:2, start_time:1727337625657064, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:25.662648] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625657790, end_time=1727337625662648, runtime=4858, *this={this:0x7f9264d850f0, type:16, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DB-0-0, dag_ret:0, dag_status:2, start_time:1727337625657064, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:25.662654] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5590, dag_cnt=2, dag_cnts_[dag.get_type()]=2, &dag=0x7f9264d80b60, dag={ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5DB-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625657064, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}) [2024-09-26 16:00:25.662660] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5DB-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625657061, task_id:YB42C0A8002D-0006230056D8C5DB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.662687] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337625620325, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.662706] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=18] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.662717] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.662720] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.662746] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101063}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.662766] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:101063}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101063}, table_count:1, [{i:0, table_key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625660345, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.38ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.662789] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E1-0-0, dag_ret:0, dag_status:2, start_time:1727337625658804, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f34a0, output_block_cnt_arr:0x7f92728f34a8, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337625660345, estimated_finish_time:1727337642662789}) [2024-09-26 16:00:25.662795] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d82e40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E1-0-0, dag_ret:0, dag_status:2, start_time:1727337625658804, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.662800] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=5] task finish process(ret=0, start_time=1727337625660343, end_time=1727337625662800, runtime=2457, *this={this:0x7f9264d82e40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E1-0-0, dag_ret:0, dag_status:2, start_time:1727337625658804, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.662810] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=3] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.662873] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101063, range=0x7f92a9b52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d2e060, count:8, allocator:0x7f92c19447d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e3e0, local_datums:0x7f9287d2e3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:25.662926] ERROR issue_dba_error (ob_log.cpp:1875) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.662928] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.662936] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=0] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337625620325, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.662953] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625662847, end_time=1727337625662953, runtime=106, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E1-0-0, dag_ret:0, dag_status:2, start_time:1727337625658804, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.662961] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4157, dag_cnt=1, dag_cnts_[dag.get_type()]=1, &dag=0x7f9264d82a80, dag={ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E1-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625658804, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.662968] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E1-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625658801, task_id:YB42C0A8002D-0006230056D8C5E1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.662996] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337625620468, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:25.663004] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:25.663021] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:25.663024] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:25.663028] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101067}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:25.663046] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101067}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101067}, table_count:1, [{i:0, table_key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b86a2d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337625661965, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.06ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:25.663072] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d849a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E7-0-0, dag_ret:0, dag_status:2, start_time:1727337625661236, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab4a0, output_block_cnt_arr:0x7f92684ab4a8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337625661965, estimated_finish_time:1727337642663072}) [2024-09-26 16:00:25.663077] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d84d60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d849a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E7-0-0, dag_ret:0, dag_status:2, start_time:1727337625661236, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.663084] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=6] task finish process(ret=0, start_time=1727337625661963, end_time=1727337625663083, runtime=1120, *this={this:0x7f9264d84d60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d849a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E7-0-0, dag_ret:0, dag_status:2, start_time:1727337625661236, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.663141] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=7] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:25.663192] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=101067, range=0x7f92aa352200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d2e060, count:8, allocator:0x7f9269ffa7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e3e0, local_datums:0x7f9287d2e3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:25.663413] ERROR issue_dba_error (ob_log.cpp:1875) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:25.663419] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:25.663428] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=0] partition merge iter row count(i=0, row_count=227, ghost_row_count=0, pkey={tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337625620468, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:25.663459] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337625663165, end_time=1727337625663459, runtime=294, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d849a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E7-0-0, dag_ret:0, dag_status:2, start_time:1727337625661236, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:25.663468] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2231, dag_cnt=0, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264d849a0, dag={ObIDag:{this:0x7f9264d849a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E7-0-0, dag_ret:-4184, dag_status:5, start_time:1727337625661236, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:25.663475] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E7-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337625661230, task_id:YB42C0A8002D-0006230056D8C5E7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:25.679903] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337625679902, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000284, add_bytes: 0 [2024-09-26 16:00:25.692464] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:25.692473] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.723745] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:25.723768] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.723887] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=14565376, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:25.723939] INFO [COMMON] wash (ob_kvcache_store.cpp:335) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Wash memory, (tenant_id=1, cache_size=14565376, lower_mem_limit=2147483648, upper_mem_limit=2147483648, min_wash_size=0, max_wash_size=0, mem_usage=1303588864, reserve_mem=137180449, wash_size=2252308480) [2024-09-26 16:00:25.724021] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash time detail, (compute_wash_size_time=68, refresh_score_time=17, wash_time=12) [2024-09-26 16:00:25.724642] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=2] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=440398, clean_num=31457) [2024-09-26 16:00:25.727633] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:25.727639] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:25.727648] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337625727637) [2024-09-26 16:00:25.727654] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337625627074, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:25.727672] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223354962151, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.727672] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337625288788]}) [2024-09-26 16:00:25.727685] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:25.727703] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:25.727708] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:25.727712] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:25.727718] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.727721] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.727723] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337625727657) [2024-09-26 16:00:25.753984] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:25.753993] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.784262] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:25.784278] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.789202] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:25.789213] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337625789199], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:25.789221] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337625789199], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:25.789225] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:25.789228] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:25.790045] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337624789084]) [2024-09-26 16:00:25.790053] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337624789084], now0=[mts=1727337624789084], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:25.790070] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337624788043, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:25.790089] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:25.790097] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337624788043, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:25.790104] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337624788043, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:25.790124] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=19][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:25.790129] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:25.790133] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:25.790136] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:25.790139] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:25.790144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=21, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:25.790149] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] will sleep(sleep_us=21000, remain_us=7761679, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:25.811608] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:21, local_retry_times:21, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:25.811636] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=22][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:25.811639] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:25.811641] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:25.811649] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:25.811692] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:25.811698] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:25.814544] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:25.814553] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.821385] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.821425] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.821430] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.821439] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.821441] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.821444] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.822368] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.822370] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.822378] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.822384] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.822391] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:25.828212] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337625789199]}) [2024-09-26 16:00:25.828218] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:25.828225] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:25.828239] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337625828216) [2024-09-26 16:00:25.828245] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337625727657, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:25.828250] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:25.828259] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:25.828262] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:25.828570] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:25.828582] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:25.828586] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:25.828609] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=22][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:25.828613] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:25.828616] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:25.828623] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=5][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:25.828626] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:25.828628] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:25.828630] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:25.828632] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:25.828635] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:25.828637] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:25.828640] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:25.828647] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:25.828650] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:25.828653] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:25.828656] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:25.828660] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:25.828663] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:25.828666] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:25.828673] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:25.828678] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:25.828680] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:25.828682] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:25.828687] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:25.828691] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:25.828694] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885732-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:25.828697] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:25.828699] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:25.828702] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:25.828705] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337625828498, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:25.828709] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:25.828712] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:25.828745] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:25.828750] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337625828749, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=494, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:25.828757] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:25.828760] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:25.828776] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223355064719, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.828781] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:25.828788] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:25.828793] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:25.828795] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:25.828800] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.828802] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.828804] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337625828764) [2024-09-26 16:00:25.844775] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:25.844784] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.859482] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337626787862, now=[mts=1727337624858067]) [2024-09-26 16:00:25.859491] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337626787862, now=[mts=1727337624858067], now0=[mts=1727337624858067], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:25.859498] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a31201a0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500666, assoc_session_id:3221500666, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337624857708, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:25.859515] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22fdff8, id:3221500666, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a31201a0}) [2024-09-26 16:00:25.859523] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500666, *tx_desc={this:0x7f92a31201a0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500666, assoc_session_id:3221500666, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337624857708, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:25.859531] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=8] start stmt(ret=-4283, auto_commit=true, session_id=3221500666, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a31201a0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500666, assoc_session_id:3221500666, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337624857708, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337624858046, use_das=false, nested_level=0, session={this:0x7f92a22fdff8, id:3221500666, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a31201a0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:25.859548] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:25.859553] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:25.859556] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:25.859558] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:25.859561] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:25.859565] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:25.859569] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=998293, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337626857862) [2024-09-26 16:00:25.859573] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:25.859580] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:25.859582] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:25.859583] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:25.859588] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:25.859607] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2169) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3] need check schema(new_schema_version=1, cached_tenant_schema_version=1, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = ? ORDER BY row_id, column_name) [2024-09-26 16:00:25.859625] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=5][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:25.859628] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:25.875099] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:25.875107] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.888750] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=4] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-09-26 16:00:25.888763] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=9] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=685) [2024-09-26 16:00:25.905400] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:25.905408] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.905419] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:25.925014] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:25.925115] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=50, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:25.925722] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=471855, clean_num=31457) [2024-09-26 16:00:25.927736] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:25.928823] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337625789199]}) [2024-09-26 16:00:25.928821] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:25.928832] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337625928817) [2024-09-26 16:00:25.928837] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337625828248, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:25.928854] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223355165318, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:25.928861] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:25.928868] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:25.928873] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:25.928876] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:25.928882] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.928884] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:25.928887] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337625928839) [2024-09-26 16:00:25.935676] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16] success to notify_need_writing_throttling True [2024-09-26 16:00:25.935681] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.964038] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:25.965922] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:25.965928] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:25.974167] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: SHARED_BLOCK_ReaderWriter_IO, mode: read, size: 4096.00, iops: 52.71, rt: 2660.58) [2024-09-26 16:00:25.974176] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:25.974195] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 12227.00) [2024-09-26 16:00:25.974198] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:25.974204] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:25.974208] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:25.989570] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] new mysql sessid created(conn.sessid_=3221500671, support_ssl=false) [2024-09-26 16:00:25.989594] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221500671, sess.client_addr_="127.0.0.1:36312") [2024-09-26 16:00:25.989600] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f9262e04030, session_id:3221500671, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:25.989618] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:25.989629] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:25.989672] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9A-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:25.989840] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9A-0-0] [lt=3][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:25.989848] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9A-0-0] [lt=7][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"{v`[*Zk*/Kma!ydia+R:"}, ret=-4043) [2024-09-26 16:00:25.989854] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9A-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:25.989858] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9A-0-0] [lt=4][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:25.989866] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9A-0-0] [lt=3] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:25.989887] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9A-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500671, proxy_sessid:0}) [2024-09-26 16:00:25.989891] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9A-0-0] [lt=3] mark session id unused(sessid=3221500671) [2024-09-26 16:00:25.989894] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9A-0-0] [lt=1][errcode=0] server close connection(sessid=3221500671, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:25.989897] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9A-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500671, proxy_sessid=0) [2024-09-26 16:00:25.989900] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9A-0-0] [lt=3][errcode=-4016] session is null [2024-09-26 16:00:25.989903] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9A-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500671, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:25.989932] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f9262e04030, session_id:3221500671, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337625988480, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:25.989941] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] kill and revert session(conn.sessid_=3221500671, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:25.989944] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9262e04030, session_id:3221500671, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337625988480, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:25.989949] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500671, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:25.996225] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:25.996231] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.020963] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:26.020975] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-09-26 16:00:26.020979] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:26.020983] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-09-26 16:00:26.020986] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:26.020989] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:26.020992] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-09-26 16:00:26.020994] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:26.020996] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:26.020998] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:26.021000] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=1][errcode=-5019] resolve table failed(ret=-5019) [2024-09-26 16:00:26.021002] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:2822) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=1][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-09-26 16:00:26.021004] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2676) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] resolve joined table failed(ret=-5019) [2024-09-26 16:00:26.021005] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:26.021007] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:26.021010] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:26.021012] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:26.021018] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:26.021020] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:26.021023] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:26.021025] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:26.021027] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-09-26 16:00:26.021029] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:26.021032] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:26.021037] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:26.021042] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:26.021044] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:26.021045] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:26.021051] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:26.021054] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:26.021056] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [3024][BlackListServic][T0][YB42C0A8002D-0006230056284F9A-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-09-26 16:00:26.021059] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:26.021061] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:26.021063] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:26.021065] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337626020865, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:26.021068] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:26.021071] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:222) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:26.021074] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:398) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2] start to print blacklist info [2024-09-26 16:00:26.021098] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=1] ls blacklist refresh finish(cost_time=461) [2024-09-26 16:00:26.023703] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EC-0-0] [lt=26][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:26.023713] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EC-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:26.023723] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EC-0-0] [lt=9][errcode=-4018] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:26.027500] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:26.027509] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.028146] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:26.028155] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:26.028159] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:26.028162] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:26.028165] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:26.028167] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:26.028169] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:26.028171] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:26.028173] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:26.028175] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:26.028177] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:26.028178] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:26.028180] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:26.028182] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:26.028187] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:26.028189] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:26.028191] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:26.028193] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:26.028195] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:26.028197] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:26.028199] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:26.028203] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:26.028208] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:26.028210] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:26.028211] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:26.028216] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:26.028219] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:26.028221] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:26.028224] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85873-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:26.028226] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:26.028228] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:26.028230] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85873-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337626028096, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:26.028233] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85873-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:26.028235] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85873-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:26.028252] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:26.028254] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85873-0-0] [lt=2][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:26.029007] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337625789199]}) [2024-09-26 16:00:26.029019] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D7F4-0-0] [lt=9][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337626029002}) [2024-09-26 16:00:26.029156] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:26.029166] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337626029154) [2024-09-26 16:00:26.029170] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337625928839, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:26.029186] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223355265100, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:26.029192] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:26.029199] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:26.029204] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:26.029208] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:26.029214] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.029216] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.029218] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337626029173) [2024-09-26 16:00:26.037606] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=8] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:26.037865] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:26.037871] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=5][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:26.037875] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:26.037879] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:26.037881] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:26.037883] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:26.037886] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:26.037888] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:26.037889] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:26.037891] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:26.037893] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:26.037894] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:26.037896] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:26.037909] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3C-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:26.057790] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:26.057802] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.061660] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4018, dropped:3, tid:2923}]) [2024-09-26 16:00:26.079249] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=8] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:26.089002] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:26.089009] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.119326] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:26.119336] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.127238] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:26.127341] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=49, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:26.127945] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=503312, clean_num=31457) [2024-09-26 16:00:26.129056] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:26.129109] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=29] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337625789199]}) [2024-09-26 16:00:26.129232] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:26.129242] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337626129229) [2024-09-26 16:00:26.129248] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337626029173, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:26.129265] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223355365577, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:26.129271] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:26.129279] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:26.129284] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:26.129287] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:26.129292] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.129294] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.129296] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337626129252) [2024-09-26 16:00:26.149595] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:26.149604] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.179857] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:26.179863] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.195466] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] ====== check clog disk timer task ====== [2024-09-26 16:00:26.195474] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=6] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:26.195481] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:26.209989] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:26.210001] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.227452] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485882-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:26.227522] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485882-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:26.229340] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337625789199]}) [2024-09-26 16:00:26.229338] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:26.229351] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337626229335) [2024-09-26 16:00:26.229357] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337626129251, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:26.229375] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223355465491, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:26.229381] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:26.229389] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:26.229394] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:26.229397] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:26.229414] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.229418] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.229429] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337626229360) [2024-09-26 16:00:26.240417] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:26.240430] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.249180] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=14] PNIO [ratelimit] time: 1727337626249180, bytes: 14580466, bw: 0.003375 MB/s, add_ts: 1000159, add_bytes: 3540 [2024-09-26 16:00:26.270686] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:26.270707] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.281483] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D356A-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:26.290213] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337626219572, now=[mts=1727337625290207]) [2024-09-26 16:00:26.290222] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=9][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337626219572, now=[mts=1727337625290207], now0=[mts=1727337625290207], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:26.290230] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926918be00, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337625290043, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:26.290247] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926918be00}) [2024-09-26 16:00:26.290256] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926918be00, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337625290043, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:26.290262] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926918be00, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337625290043, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337624289801, use_das=false, nested_level=0, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926918be00}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:26.290280] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:26.290285] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:26.290288] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:26.290291] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:26.290293] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:26.290297] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:26.290304] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=6][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:26.290311] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:26.290313] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:26.290314] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:26.290319] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:26.290322] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000521) [2024-09-26 16:00:26.290319] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.290325] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:26.290327] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337626290316], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:26.290334] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337626290316], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:26.290329] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:26.290338] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=5124, get_gts_with_stc_cnt=12039, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-09-26 16:00:26.290342] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=12][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:26.290342] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337626290316]}) [2024-09-26 16:00:26.290345] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:26.290348] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:26.290353] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:26.290382] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:26.290384] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:26.290386] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:26.290389] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:26.290392] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:26.290395] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:26.290527] INFO [SQL] check_session_leak (ob_sql_session_mgr.cpp:597) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=2] get current session count(used_session_count=2, hold_session_count=2, session_leak_count_threshold=100) [2024-09-26 16:00:26.290532] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C75-0-0] [lt=4][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:26.290538] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:26.290540] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:26.290555] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:26.290559] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:26.290895] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:26.290903] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=8][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:26.300963] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:26.300970] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.328391] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:26.328511] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=55, refresh_score_time=30, wash_time=2) [2024-09-26 16:00:26.329126] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=534769, clean_num=31457) [2024-09-26 16:00:26.329241] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:26.330248] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337626290316]}) [2024-09-26 16:00:26.330265] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:26.330282] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337626330262) [2024-09-26 16:00:26.330288] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337626229360, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:26.330305] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:26.330314] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:26.330319] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:26.330673] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885733-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:26.330684] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885733-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:26.330711] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885733-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:26.330744] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337626330743, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=433, wlock_time=11, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:26.330752] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:26.330768] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223355566730, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:26.330773] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:26.330779] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:26.330784] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:26.330787] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:26.330792] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.330794] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.330796] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337626330756) [2024-09-26 16:00:26.332206] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:26.332211] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.362505] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:26.362512] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.382984] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=5] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:26.392763] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:26.392771] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.423078] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:26.423084] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.430843] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337626290316]}) [2024-09-26 16:00:26.430846] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:26.430856] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337626430843) [2024-09-26 16:00:26.430861] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337626330302, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:26.430878] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223355667342, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:26.430884] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:26.430891] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:26.430896] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:26.430898] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:26.430904] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.430907] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.430909] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337626430864) [2024-09-26 16:00:26.443539] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.443551] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.443553] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.443557] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.443558] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.443558] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.443562] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.443562] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.443562] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.444543] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.444553] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.444555] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.444557] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.444548] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.444560] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.444572] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.444585] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=12][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.444585] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=24][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.445492] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.445501] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.445506] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.446486] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.446493] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.446505] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.446508] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.446497] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.446513] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=16][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.446506] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.446520] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.446522] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.447468] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.447469] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.447471] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.447472] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.447474] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.447475] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.447469] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.447480] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.447483] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.448404] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.448461] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.448463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=58][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.448464] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.448465] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.448466] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.448470] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.448481] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.448484] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:26.449424] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.449435] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.449436] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.450404] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.450406] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.450412] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.451342] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.451342] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.451342] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.452341] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.452343] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.452348] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.453346] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.453364] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:26.453362] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.453372] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.453369] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.454322] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.454327] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.454326] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.455286] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.455290] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.455303] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=26][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.456240] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.456256] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.456263] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.457222] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.458198] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.458199] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.458207] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.459176] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.459177] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.459177] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.460140] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.460168] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.460169] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.461142] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.461153] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.461159] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.462132] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.462152] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.462152] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.463084] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.463094] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.463094] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.464056] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.464071] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.464083] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.464081] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:100, tid:3001}]) [2024-09-26 16:00:26.465023] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.465031] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.466062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.466063] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.466066] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.466986] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.466987] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.467986] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.467984] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.467973] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.468945] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.468945] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.468946] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=40][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.469911] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.469922] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.469934] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.470882] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.470892] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.470896] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.471874] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.471876] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.471881] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.473811] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.473815] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.473817] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.474800] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.474807] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.474800] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.475767] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.475774] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.475783] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.476755] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.476764] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.476774] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.477716] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.477744] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.477753] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.478703] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:26.483578] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:26.483585] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.513860] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:26.513867] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.523101] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858ED-0-0] [lt=0][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:26.523114] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858ED-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:26.523124] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858ED-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:26.523130] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858ED-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:26.529291] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:26.529400] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash time detail, (compute_wash_size_time=54, refresh_score_time=17, wash_time=2) [2024-09-26 16:00:26.529884] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:26.530049] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=566226, clean_num=31457) [2024-09-26 16:00:26.530934] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:26.530946] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337626530931) [2024-09-26 16:00:26.530951] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337626430864, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:26.530954] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=25] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337626290316]}) [2024-09-26 16:00:26.530971] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223355766428, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:26.530977] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:26.530997] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:26.531003] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:26.531006] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:26.531012] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.531014] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.531016] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337626530954) [2024-09-26 16:00:26.538162] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6291, mem_used=10398720) [2024-09-26 16:00:26.544146] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:00:26.544156] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.559810] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=8] gc stale ls task succ [2024-09-26 16:00:26.574460] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:26.574468] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.600807] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.600822] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.600830] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.601762] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.604725] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:26.604742] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.612528] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:26.631069] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337626290316]}) [2024-09-26 16:00:26.631070] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:26.631081] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337626631066) [2024-09-26 16:00:26.631087] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337626530954, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:26.631073] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=25] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:26.631121] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223355867546, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:26.631129] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:26.631136] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:26.631142] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:26.631145] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:26.631152] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.631154] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.631156] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337626631091) [2024-09-26 16:00:26.634971] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:26.634991] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.652581] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=42] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:26.653545] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=5] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:26.665258] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:26.665267] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.665295] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4184, dropped:500, tid:2885}]) [2024-09-26 16:00:26.686751] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1727337626686750, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006848, add_bytes: 0 [2024-09-26 16:00:26.695531] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:26.695539] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.725827] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:26.725833] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.730787] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:26.730887] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=48, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:26.730969] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=629120, replace_num=15728) [2024-09-26 16:00:26.730977] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:26.731498] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=597683, clean_num=31457) [2024-09-26 16:00:26.731649] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:26.731659] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337626731646) [2024-09-26 16:00:26.731678] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337626631090, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:26.731695] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337626290316]}) [2024-09-26 16:00:26.731705] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223355968009, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:26.731712] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:26.731720] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:26.731725] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:26.731727] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:26.731734] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.731736] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.731738] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337626731682) [2024-09-26 16:00:26.756111] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:26.756124] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.786398] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:26.786421] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.791407] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:26.791421] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337626791403], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:26.791430] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337626791403], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:26.791434] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:26.791439] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:26.801039] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [2915][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=7] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-09-26 16:00:26.810935] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337625811690]) [2024-09-26 16:00:26.810947] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337625811690], now0=[mts=1727337625811690], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:26.810954] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337625810709, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:26.810973] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=18][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:26.810982] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337625810709, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:26.810990] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337625810709, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:26.811009] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=18][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:26.811014] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:26.811017] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:26.811019] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:26.811022] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:26.811025] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=22, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:26.811031] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] will sleep(sleep_us=22000, remain_us=6740798, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:26.816625] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:26.816633] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.820608] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=47] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.820618] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.820630] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.821540] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.821549] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.822494] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.822538] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.822559] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.822585] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.822587] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.822597] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:26.831679] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:26.831699] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:26.831703] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:26.832058] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:26.832080] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=20][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:26.832085] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:26.832088] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:26.832092] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:26.832095] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:26.832098] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:26.832101] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:26.832104] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:26.832107] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:26.832109] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:26.832112] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:26.832117] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:26.832119] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:26.832127] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:26.832129] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:26.832134] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:26.832136] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:26.832140] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:26.832144] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=4][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:26.832148] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:26.832154] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:26.832161] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:26.832164] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:26.832166] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:26.832173] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:26.832177] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:26.832180] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885734-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:26.832183] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:26.832186] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:26.832189] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:26.832192] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337626831973, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:26.832196] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:26.832199] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:26.832242] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:26.832247] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337626832246, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=553, wlock_time=12, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:26.832254] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:26.832258] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:26.832281] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223356068567, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:26.832287] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:26.832294] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:26.832299] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:26.832302] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:26.832312] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.832315] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.832316] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337626832263) [2024-09-26 16:00:26.832448] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337626791403]}) [2024-09-26 16:00:26.833100] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:22, local_retry_times:22, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:26.833117] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:26.833121] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:26.833123] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:26.833128] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:26.833165] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:26.833168] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:26.846931] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:26.846958] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.858695] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=20][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337626787862, now=[mts=1727337625859625]) [2024-09-26 16:00:26.858719] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=23][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337626787862, now=[mts=1727337625859625], now0=[mts=1727337625859625], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:26.858727] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a31201a0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500666, assoc_session_id:3221500666, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337625858582, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:26.858747] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=18][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22fdff8, id:3221500666, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a31201a0}) [2024-09-26 16:00:26.858755] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500666, *tx_desc={this:0x7f92a31201a0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500666, assoc_session_id:3221500666, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337625858582, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:26.858763] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500666, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a31201a0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500666, assoc_session_id:3221500666, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337625858582, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337624858046, use_das=false, nested_level=0, session={this:0x7f92a22fdff8, id:3221500666, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a31201a0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:26.858783] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=19][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:26.858788] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:26.858791] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:26.858794] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:26.858796] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:26.858801] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:26.858807] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:26.858814] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:26.858816] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:26.858818] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:26.858824] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:26.858827] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000780) [2024-09-26 16:00:26.858830] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507D-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:26.858833] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:26.858836] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:26.858839] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:26.858842] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8340, start=1727337624858042, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:26.858846] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:00:26.858849] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:26.858890] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:26.858895] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:26.858897] WDIAG [SHARE] get_snapshot_gc_scn (ob_global_stat_proxy.cpp:164) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:26.858900] WDIAG [STORAGE] get_global_info (ob_tenant_freeze_info_mgr.cpp:811) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] fail to get global info(ret=-4012, tenant_id=1) [2024-09-26 16:00:26.858903] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:954) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] failed to get global info(ret=-4012) [2024-09-26 16:00:26.858906] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1008) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to try update info(tmp_ret=-4012, tmp_ret="OB_TIMEOUT") [2024-09-26 16:00:26.877221] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:26.877238] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.908516] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24] success to notify_need_writing_throttling True [2024-09-26 16:00:26.908530] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.908545] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:26.931004] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=4] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:26.931917] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:26.931927] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:26.931942] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337626931913) [2024-09-26 16:00:26.931947] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337626731682, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:26.931970] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223356167555, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:26.931976] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:26.931986] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:26.931991] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:26.931994] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:26.932005] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.932021] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:26.932024] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337626931950) [2024-09-26 16:00:26.932034] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:26.932137] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=69, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:26.932473] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337626791403]}) [2024-09-26 16:00:26.932792] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=629140, clean_num=31457) [2024-09-26 16:00:26.938749] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:26.938756] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.964163] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] tx gc loop thread is running(MTL_ID()=1) [2024-09-26 16:00:26.964197] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=33] try gc retain ctx [2024-09-26 16:00:26.964599] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-09-26 16:00:26.964631] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:26.969053] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:26.969063] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:26.979778] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10689.00) [2024-09-26 16:00:26.979787] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:26.979801] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:26.979806] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:26.990810] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] new mysql sessid created(conn.sessid_=3221500675, support_ssl=false) [2024-09-26 16:00:26.990849] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221500675, sess.client_addr_="127.0.0.1:36314") [2024-09-26 16:00:26.990859] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500675, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:26.990882] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:26.990891] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:26.990933] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9A-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:26.991126] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9A-0-0] [lt=7][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:26.991137] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9A-0-0] [lt=10][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"^)R?&>G+@>Usbo(xE1c!"}, ret=-4043) [2024-09-26 16:00:26.991154] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9A-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:26.991159] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9A-0-0] [lt=5][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:26.991168] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9A-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:26.991192] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9A-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500675, proxy_sessid:0}) [2024-09-26 16:00:26.991196] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9A-0-0] [lt=4] mark session id unused(sessid=3221500675) [2024-09-26 16:00:26.991201] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9A-0-0] [lt=3][errcode=0] server close connection(sessid=3221500675, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:26.991204] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9A-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500675, proxy_sessid=0) [2024-09-26 16:00:26.991207] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9A-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:26.991210] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9A-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500675, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:26.991292] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500675, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337626990597, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:26.991310] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] kill and revert session(conn.sessid_=3221500675, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:26.991314] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500675, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337626990597, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:26.991319] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500675, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:26.999306] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:26.999314] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.028938] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:27.028951] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:27.028956] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:27.028959] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:27.028964] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:27.028966] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:27.028970] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:27.028973] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:27.028975] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:27.028977] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:27.028979] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:27.028981] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:27.028983] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:27.028986] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:27.028992] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:27.028994] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:27.028997] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:27.028999] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:27.029001] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:27.029004] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:27.029006] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:27.029011] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:27.029030] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:27.029033] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:27.029035] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:27.029044] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:27.029047] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:27.029051] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85874-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:27.029053] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:27.029056] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85874-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:27.029061] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85874-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:27.029064] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337627028866, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:27.029067] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85874-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:27.029070] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85874-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:27.029095] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85874-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:27.029100] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85874-0-0] [lt=4][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:27.029173] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EE-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:27.029182] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EE-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:27.029190] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EE-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:27.029195] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EE-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:27.029521] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:27.029529] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.032555] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337626791403]}) [2024-09-26 16:00:27.032566] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:27.032574] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337627032562) [2024-09-26 16:00:27.032572] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D800-0-0] [lt=5][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337627032550}) [2024-09-26 16:00:27.032578] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337626931950, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:27.032596] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223356269064, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:27.032602] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:27.032609] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:27.032614] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:27.032617] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:27.032624] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.032626] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.032628] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337627032581) [2024-09-26 16:00:27.038380] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=5] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:27.038704] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:27.038716] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:27.038719] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:27.038723] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:27.038725] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:27.038727] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:27.038731] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:27.038733] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:27.038735] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:27.038736] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:27.038738] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:27.038740] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:27.038741] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:27.038745] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:27.038751] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:27.038754] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:27.038756] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:27.038759] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:27.038761] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:27.038764] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:27.038766] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:27.038771] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:27.038776] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:27.038778] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:27.038779] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:27.038785] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:27.038788] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:27.038791] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F3E-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:27.038794] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:27.038797] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:27.038799] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:27.038802] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337627038656, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:27.038805] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:27.038852] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:27.059890] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:27.059907] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.079423] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=7] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:27.090135] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:27.090144] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.120399] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:27.120409] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.131149] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:27.133148] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337626791403]}) [2024-09-26 16:00:27.133170] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:27.133179] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337627133167) [2024-09-26 16:00:27.133186] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337627032581, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:27.133203] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223356369647, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:27.133210] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:27.133217] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:27.133222] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:27.133226] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:27.133232] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.133235] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.133237] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337627133189) [2024-09-26 16:00:27.133253] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:27.133370] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=57, refresh_score_time=22, wash_time=2) [2024-09-26 16:00:27.133994] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=660597, clean_num=31457) [2024-09-26 16:00:27.150676] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29] success to notify_need_writing_throttling True [2024-09-26 16:00:27.150682] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.180990] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:27.181017] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.199714] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:27.199724] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:27.199729] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:27.211232] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:27.211238] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.216117] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337626791403]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:27.216129] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:27.228183] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:27.228194] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:27.228198] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:27.228201] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:27.228204] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:27.228206] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:27.228209] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:27.228212] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:27.228214] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:27.228215] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:27.228217] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:27.228220] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:27.228232] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:27.228234] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:27.228240] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:27.228243] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:27.228245] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:27.228247] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:27.228250] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:27.228252] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:27.228254] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:27.228259] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:27.228265] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:27.228267] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:27.228268] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:27.228274] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:27.228277] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:27.228280] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:27.228282] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:27.228284] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:27.228286] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:27.228289] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337627228118, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:27.228292] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:27.228295] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:27.228322] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:27.228324] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:27.228326] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:27.228328] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:27.228331] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:27.228334] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:27.228335] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485883-0-0] [lt=1][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:27.233314] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337626791403]}) [2024-09-26 16:00:27.233386] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:27.233394] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337627233383) [2024-09-26 16:00:27.233398] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337627133189, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:27.233412] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223356469207, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:27.233418] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:27.233424] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:27.233428] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:27.233431] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:27.233435] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.233437] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.233439] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337627233401) [2024-09-26 16:00:27.241506] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:27.241512] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.256165] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337627256164, bytes: 14584288, bw: 0.003620 MB/s, add_ts: 1006984, add_bytes: 3822 [2024-09-26 16:00:27.271850] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:27.271857] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.284494] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=6][errcode=-4018] try free send task failed(ret=-4018) [2024-09-26 16:00:27.284822] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D35C8-0-0] [lt=13][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:27.290395] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337628220564, now=[mts=1727337626290894]) [2024-09-26 16:00:27.290402] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337628220564, now=[mts=1727337626290894], now0=[mts=1727337626290894], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:27.290409] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926918dba0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337626290204, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:27.290425] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926918dba0}) [2024-09-26 16:00:27.290433] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926918dba0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337626290204, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:27.290440] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926918dba0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337626290204, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337626290865, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926918dba0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:27.290456] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:27.290461] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:27.290464] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:27.290466] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:27.290468] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:27.290471] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:27.290475] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=1000089, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337628290564) [2024-09-26 16:00:27.290478] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:27.290484] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:27.290486] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:27.290488] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:27.290493] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:27.290526] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=1][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:27.290528] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:27.291988] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.291998] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337627291985], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:27.292005] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337627291985], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:27.302066] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:27.302074] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.331393] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:27.332343] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:27.332349] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.334337] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337627291985]}) [2024-09-26 16:00:27.334347] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:27.334392] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337627334342) [2024-09-26 16:00:27.334399] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337627233401, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:27.334406] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:27.334417] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:27.334421] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:27.334497] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:27.334626] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=61, refresh_score_time=27, wash_time=3) [2024-09-26 16:00:27.334837] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:27.334846] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:27.334851] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:27.334856] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:27.334860] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:27.334863] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:27.334868] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:27.334871] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:27.334874] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:27.334876] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:27.334879] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:27.334882] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:27.334885] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:27.334888] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:27.334896] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:27.334899] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:27.334902] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:27.334905] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:27.334908] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:27.334912] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:27.334915] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:27.334922] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:27.334929] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:27.334932] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:27.334934] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:27.334941] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:27.334946] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=4][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:27.334949] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885735-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:27.334953] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:27.334956] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:27.334959] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:27.334962] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337627334745, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:27.334966] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:27.334969] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:27.335004] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:27.335008] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337627335007, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=595, wlock_time=11, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:27.335016] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:27.335019] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:27.335039] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223356570803, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:27.335045] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:27.335053] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:27.335058] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:27.335061] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:27.335067] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.335070] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.335073] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337627335025) [2024-09-26 16:00:27.335394] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=692054, clean_num=31457) [2024-09-26 16:00:27.362610] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:27.362618] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.380208] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=6] current monitor number(seq_=-1) [2024-09-26 16:00:27.383123] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=19] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:27.385099] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [2645][Blacklist][T0][Y0-0000000000000000-0-0] [lt=10] blacklist_loop exec finished(cost_time=9, is_enabled=true, send_cnt=0) [2024-09-26 16:00:27.392953] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:27.392959] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.421236] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] ====== tenant freeze timer task ====== [2024-09-26 16:00:27.421259] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=19] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:27.421291] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337625619356, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337625619319, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337625619373, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:27.421317] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:27.421382] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D2-0-0] [lt=2] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:27.421392] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D2-0-0] [lt=6] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:27.421409] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D2-0-0] [lt=15] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:27.421413] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D2-0-0] [lt=3] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:27.421416] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D2-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:27.421420] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D2-0-0] [lt=3][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:27.421422] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D2-0-0] [lt=1][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:27.421431] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D2-0-0] [lt=9] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:27.421433] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D2-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:27.421435] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D2-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:27.421438] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D2-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:27.421484] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851D2-0-0] [lt=5] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:27.423171] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:27.423179] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.434672] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:27.434695] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337627434669) [2024-09-26 16:00:27.434701] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337627334402, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:27.434719] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223356670121, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:27.434726] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:27.434746] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:27.434752] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:27.434755] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:27.434761] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.434764] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.434765] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337627434704) [2024-09-26 16:00:27.434858] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337627291985]}) [2024-09-26 16:00:27.443706] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=22] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.443718] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.443725] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.444245] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.444254] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.444260] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.444807] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.444826] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.444831] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.445390] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.445396] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.445399] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.446130] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.446137] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.446141] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.446213] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.446218] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.446222] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.446791] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.446798] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.446801] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.447239] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.447247] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.447251] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.447357] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.447361] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.447363] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.448186] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.448190] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.448192] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.448355] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.448358] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.448360] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.448796] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.448805] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.448809] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.449303] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.449309] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.449312] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.449388] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.449394] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.449397] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.449868] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.449871] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.449874] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.450141] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.450150] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.450153] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:27.450624] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.450705] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.451477] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.451533] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.452286] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.452306] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.453063] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.453083] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.453239] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:27.453247] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.453601] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.453608] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.454191] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.454241] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.454759] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.454837] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.455396] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.455483] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.456388] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.456393] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.457368] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.457384] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.458325] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.458325] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.459279] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.459307] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.460308] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.460306] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.460903] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.460935] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.461639] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.461640] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.462200] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.462201] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.463212] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.463218] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.464208] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.464211] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.465203] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.465250] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.466146] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.466180] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.467112] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.467130] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.468115] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.468115] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.469116] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.469124] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.470056] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.470062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.470584] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.470585] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.471419] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.471447] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.472021] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.472025] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.473008] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.473010] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.473638] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.473660] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.474952] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.474953] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.475921] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.475949] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.476922] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.476926] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.477872] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.477891] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.478872] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.478886] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.479831] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.479866] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.480779] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.480806] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.481781] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.481784] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.482776] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.482787] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.483723] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.483746] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:27.483753] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.483776] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.484710] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.484740] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.485354] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.485386] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.486662] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.486711] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:27.514066] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:27.514084] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.522906] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EF-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:27.522916] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EF-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:27.522925] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EF-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:27.522930] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858EF-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:27.531563] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:27.535531] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337627291985]}) [2024-09-26 16:00:27.535532] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:27.535544] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337627535529) [2024-09-26 16:00:27.535572] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337627434704, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:27.535594] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] get wrs ts(ls_id={id:1}, delta=223356771997, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:27.535600] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:27.535609] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:27.535614] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:27.535617] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:27.535623] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.535625] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.535627] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337627535578) [2024-09-26 16:00:27.535657] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:27.535757] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=48, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:27.536562] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=723511, clean_num=31457) [2024-09-26 16:00:27.538235] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6299, mem_used=10398720) [2024-09-26 16:00:27.544181] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:27.544191] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.559876] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:00:27.574683] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=39] success to notify_need_writing_throttling True [2024-09-26 16:00:27.574693] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.579537] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:45) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=0] dump dtl interm result cost(us)(dump_cost=13755, ret=0, interm count=0, dump count=342) [2024-09-26 16:00:27.593480] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:60) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11] clear dtl interm result cost(us)(clear_cost=13931, ret=0, expire_keys_.count()=0, dump count=342, interm count=0, clean count=0) [2024-09-26 16:00:27.593870] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:27.593883] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:27.593887] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:27.593890] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:27.593895] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:27.593897] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:27.593900] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:27.593903] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:27.593906] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:27.593908] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:27.593910] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:27.593913] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:27.593916] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:27.593919] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:27.593925] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:27.593928] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:27.593931] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:27.593945] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=13][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:27.593948] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:27.593951] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:27.593954] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:27.593960] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:27.593967] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=6][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:27.593976] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD0-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:27.601910] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.601926] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.601933] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.601946] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.604888] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:27.604894] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.618533] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:27.636143] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:27.636151] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.636161] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337627291985]}) [2024-09-26 16:00:27.636164] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:27.636175] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337627636161) [2024-09-26 16:00:27.636180] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337627535578, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:27.636198] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] get wrs ts(ls_id={id:1}, delta=223356872659, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:27.636205] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:27.636213] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:27.636218] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:27.636221] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:27.636227] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.636230] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.636232] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337627636186) [2024-09-26 16:00:27.636239] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=29] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:27.653699] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:27.663509] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855D9-0-0] [lt=6] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:27.666403] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:27.666410] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.694781] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1727337627694780, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008030, add_bytes: 0 [2024-09-26 16:00:27.696755] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=51] success to notify_need_writing_throttling True [2024-09-26 16:00:27.696761] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.726966] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:27.726972] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.732200] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:27.732791] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=4] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:27.736736] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:27.736742] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337627291985]}) [2024-09-26 16:00:27.736748] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337627736732) [2024-09-26 16:00:27.736754] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337627636186, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:27.736772] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223356972230, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:27.736778] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:27.736786] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:27.736792] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:27.736795] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:27.736802] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.736804] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.736807] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337627736759) [2024-09-26 16:00:27.736832] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:27.736945] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=51, refresh_score_time=27, wash_time=1) [2024-09-26 16:00:27.737569] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=754968, clean_num=31457) [2024-09-26 16:00:27.758193] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:27.758202] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.788530] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:27.788540] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.792503] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:27.792516] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337627792500], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:27.792540] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337627792500], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:27.792544] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:27.792549] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:27.819778] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:27.819788] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.820964] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.820976] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.820984] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.821363] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.821377] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.821496] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.821509] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.821679] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.821692] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.821697] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.821701] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:27.833432] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=22][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337626833164]) [2024-09-26 16:00:27.833442] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337626833164], now0=[mts=1727337626833164], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:27.833448] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337626832091, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:27.833465] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:27.833473] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337626832091, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:27.833480] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337626832091, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:27.833497] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:27.833503] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:27.833505] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:27.833508] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:27.833511] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:27.833515] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=23, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:27.833520] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] will sleep(sleep_us=23000, remain_us=5718308, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:27.836950] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337627792500]}) [2024-09-26 16:00:27.836968] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:27.836977] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337627836956) [2024-09-26 16:00:27.836983] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337627736759, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:27.836987] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:27.836995] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:27.836999] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:27.837342] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885736-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:27.837350] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885736-0-0] [lt=7][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:27.837395] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885736-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:27.837432] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337627837432, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=440, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:27.837440] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:27.837455] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223357073112, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:27.837460] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:27.837465] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:27.837470] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:27.837472] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:27.837477] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.837479] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.837481] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337627837444) [2024-09-26 16:00:27.850004] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:27.850016] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.856605] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:23, local_retry_times:23, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:27.856622] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:27.856626] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:27.856628] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:27.856633] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:27.856677] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:27.856681] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:27.872519] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:48, tid:2932}]) [2024-09-26 16:00:27.880291] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:27.880302] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.910560] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:27.910567] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.910579] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:27.932262] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:27.937156] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337627792500]}) [2024-09-26 16:00:27.937160] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:27.937176] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:27.937188] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337627937158) [2024-09-26 16:00:27.937193] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337627836985, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:27.937210] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223357173656, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:27.937216] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:27.937223] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:27.937228] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:27.937231] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:27.937236] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.937238] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:27.937240] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337627937196) [2024-09-26 16:00:27.937724] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:27.937831] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=54, refresh_score_time=16, wash_time=2) [2024-09-26 16:00:27.938449] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=786425, clean_num=31457) [2024-09-26 16:00:27.940800] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:27.940807] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.965249] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:27.971104] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:00:27.971120] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:27.987788] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 572.00) [2024-09-26 16:00:27.987796] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:27.987808] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:27.987811] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:27.991865] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500679, support_ssl=false) [2024-09-26 16:00:27.991915] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221500679, sess.client_addr_="127.0.0.1:36316") [2024-09-26 16:00:27.991922] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f9263204030, session_id:3221500679, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:27.991945] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:27.991953] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:27.991991] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9B-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:27.992171] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9B-0-0] [lt=3][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:27.992178] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9B-0-0] [lt=5][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"tnJe?K;%Fq'jc2mQ,'~h"}, ret=-4043) [2024-09-26 16:00:27.992186] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9B-0-0] [lt=6][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:27.992189] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9B-0-0] [lt=2][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:27.992196] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9B-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:27.992225] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9B-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500679, proxy_sessid:0}) [2024-09-26 16:00:27.992230] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9B-0-0] [lt=5] mark session id unused(sessid=3221500679) [2024-09-26 16:00:27.992234] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9B-0-0] [lt=2][errcode=0] server close connection(sessid=3221500679, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:27.992237] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9B-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500679, proxy_sessid=0) [2024-09-26 16:00:27.992240] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9B-0-0] [lt=3][errcode=-4016] session is null [2024-09-26 16:00:27.992243] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9B-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500679, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:27.992279] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f9263204030, session_id:3221500679, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337627991669, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:27.992287] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500679, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:27.992299] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f9263204030, session_id:3221500679, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337627991669, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:27.992304] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500679, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:28.002392] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:28.002399] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.026984] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F0-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:28.026996] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F0-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:28.027005] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F0-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:28.027011] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F0-0-0] [lt=6][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:28.030013] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:28.030027] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:28.030032] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:28.030035] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:28.030039] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:28.030041] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:28.030045] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:28.030048] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:28.030050] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:28.030051] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:28.030054] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:28.030056] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:28.030058] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:28.030060] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:28.030066] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:28.030069] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:28.030071] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:28.030073] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:28.030076] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:28.030078] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:28.030080] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:28.030099] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:28.030106] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:28.030109] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:28.030111] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:28.030116] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:28.030120] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85875-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:28.030124] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85875-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:28.030127] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:28.030130] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:28.030132] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85875-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:28.030135] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337628029944, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:28.030139] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85875-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:28.030141] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85875-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:28.030183] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85875-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:28.030186] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85875-0-0] [lt=3][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:28.032646] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:28.032656] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.037578] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337627792500]}) [2024-09-26 16:00:28.037595] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D80C-0-0] [lt=6][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337628037572}) [2024-09-26 16:00:28.037627] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:28.037635] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337628037625) [2024-09-26 16:00:28.037640] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337627937196, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:28.037657] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223357274014, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:28.037662] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:28.037669] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:28.037673] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:28.037676] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:28.037682] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.037684] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.037686] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337628037643) [2024-09-26 16:00:28.039482] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=26] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:28.039767] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:28.039775] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:28.039779] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:28.039782] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:28.039785] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:28.039787] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:28.039790] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:28.039792] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:28.039793] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:28.039795] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:28.039797] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:28.039799] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:28.039800] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:28.039802] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:28.039807] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:28.039809] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:28.039811] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:28.039813] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:28.039815] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:28.039817] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:28.039819] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:28.039823] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:28.039828] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:28.039829] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:28.039831] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:28.039835] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:28.039838] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:28.039840] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F40-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:28.039843] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:28.039845] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:28.039846] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:28.039849] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337628039712, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:28.039851] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:28.039891] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2580][RSAsyncTask1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:28.062942] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25] success to notify_need_writing_throttling True [2024-09-26 16:00:28.062964] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.079532] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=26] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:28.093294] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:28.093302] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.123486] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:28.123494] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.132661] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:28.138156] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:28.138174] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337628138153) [2024-09-26 16:00:28.138163] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337627792500]}) [2024-09-26 16:00:28.138180] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337628037643, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:28.138198] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223357374601, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:28.138204] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:28.138212] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:28.138217] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:28.138220] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:28.138226] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.138229] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.138230] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337628138183) [2024-09-26 16:00:28.139220] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:28.139377] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:28.139987] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=817882, clean_num=31457) [2024-09-26 16:00:28.153775] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:28.153780] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.184119] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:28.184126] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.196757] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=2] ====== check clog disk timer task ====== [2024-09-26 16:00:28.196768] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=8] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:28.196777] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:28.214353] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:28.214360] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.229322] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:28.229336] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:28.229341] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:28.229344] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:28.229348] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:28.229350] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:28.229354] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:28.229356] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:28.229358] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:28.229359] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:28.229362] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:28.229364] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:28.229366] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:28.229369] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:28.229375] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:28.229378] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:28.229380] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:28.229382] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:28.229385] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:28.229388] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:28.229392] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:28.229397] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:28.229403] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:28.229406] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:28.229410] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:28.229417] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:28.229431] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:28.229435] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:28.229439] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:28.229441] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:28.229443] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:28.229447] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337628229243, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:28.229450] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:28.229453] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:28.229500] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=3][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:28.229503] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=3][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:28.229506] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=3][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:28.229509] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:28.229514] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=4][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:28.229517] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:28.229518] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485884-0-0] [lt=1][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:28.238216] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:28.238226] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337628238213) [2024-09-26 16:00:28.238231] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337628138183, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:28.238249] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223357474313, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:28.238255] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:28.238276] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:28.238282] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:28.238285] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:28.238291] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.238294] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.238295] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337628238234) [2024-09-26 16:00:28.238307] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337627792500]}) [2024-09-26 16:00:28.244600] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:28.244606] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.264185] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337628264184, bytes: 14587828, bw: 0.003349 MB/s, add_ts: 1008020, add_bytes: 3540 [2024-09-26 16:00:28.274964] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:28.274985] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.287651] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D3611-0-0] [lt=12][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:28.291453] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337628220564, now=[mts=1727337627290525]) [2024-09-26 16:00:28.291468] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=14][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337628220564, now=[mts=1727337627290525], now0=[mts=1727337627290525], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:28.291477] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926918dba0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337627290387, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:28.291500] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=22][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926918dba0}) [2024-09-26 16:00:28.291511] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=11][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926918dba0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337627290387, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:28.291522] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=10] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926918dba0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337627290387, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337626290865, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926918dba0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:28.291555] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=32][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:28.291563] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=7][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:28.291567] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:28.291571] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:28.291574] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:28.291580] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:28.291586] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5] already timeout, do not need sleep(sleep_us=-1022, remain_us=-1022, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, timeout_timestamp=1727337628290564) [2024-09-26 16:00:28.291591] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:28.291601] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=6][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:28.291604] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:28.291606] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:28.291613] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:28.291652] WDIAG [SQL] open_plan (ob_result_set.cpp:131) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4012] query is timeout(ret=-4012, timeout_ts=1727337628290564, start_time=1727337626290865) [2024-09-26 16:00:28.291679] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=27][errcode=-4012] execute plan failed(ret=-4012) [2024-09-26 16:00:28.291683] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4012] open result set failed(ret=-4012) [2024-09-26 16:00:28.291688] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:28.291692] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:28.291695] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:28.291699] WDIAG [SQL] do_close_plan (ob_result_set.cpp:760) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4006] exec result is null(ret=-4006) [2024-09-26 16:00:28.291703] WDIAG [SQL] do_close (ob_result_set.cpp:855) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2024-09-26 16:00:28.291707] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:28.291709] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:28.291714] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-09-26 16:00:28.291718] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C76-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:28.291722] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=3] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000856) [2024-09-26 16:00:28.291725] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:28.291729] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:28.291734] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:28.291737] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:28.291741] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:28.291748] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=5][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:28.291775] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:28.291779] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:28.291783] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:28.291787] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:28.291793] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=4][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:28.291797] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:28.291934] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C76-0-0] [lt=3][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:28.291945] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:28.291948] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:28.291968] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=3] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:28.291974] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=5][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:28.292342] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:28.292361] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=18][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:28.293562] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.293573] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337628293559], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:28.293580] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337628293559], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:28.305203] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:28.305210] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.313950] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD1-0-0] [lt=2][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-09-26 16:00:28.313961] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD1-0-0] [lt=11][errcode=-4002] refresh sever list failed(ret=-4002) [2024-09-26 16:00:28.313964] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD1-0-0] [lt=3][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-09-26 16:00:28.320806] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2648) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] [PALF_DUMP](palf_id=1, self="192.168.0.45:2882", [SlidingWindow]={palf_id:1, self:"192.168.0.45:2882", lsn_allocator:{max_log_id:183389608, max_lsn:629872191939, max_scn:{val:1727114270763524000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:629872191817}, reuse_lsn:{lsn:629872191939}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:629872191939}}, last_submit_lsn:{lsn:629872191817}, last_submit_end_lsn:{lsn:629872191939}, last_submit_log_id:183389608, last_submit_log_pid:19, max_flushed_lsn:{lsn:629872191817}, max_flushed_end_lsn:{lsn:629872191939}, max_flushed_log_pid:19, committed_end_lsn:{lsn:629872191939}, last_slide_log_id:183389608, last_slide_scn:{val:1727114270763524000, v:0}, last_slide_lsn:{lsn:629872191817}, last_slide_end_lsn:{lsn:629872191939}, last_slide_log_pid:19, last_slide_log_accum_checksum:2487403767, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f92b3ff0530}, [StateMgr]={this:0x7f92b3ff42b0, self:"192.168.0.45:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:374}, leader:"192.168.0.45:2882", leader_epoch:195, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:629872191939}, scan_disk_log_finished:true, last_check_start_id:183389609, is_changing_config_with_arb:false, reconfirm_start_time_us:1727336098224413, palf_role_change_cb:0x7f92b3ffab70, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.0.45:2882", alive_paxos_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:374, prev:{version:1, config:{config_version:{proposal_id:373, config_seq:374}, log_sync_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:374, config_seq:375}, log_sync_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:19, prev_lsn:{lsn:629872191817}, prev_mode_pid:374}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:19, prev_lsn:{lsn:629872191817}, prev_end_lsn:{lsn:629872191939}, prev_mode_pid:374}, reconfig_barrier:{prev_log_proposal_id:19, prev_lsn:{lsn:629872191817}, prev_end_lsn:{lsn:629872191939}, prev_mode_pid:374}, persistent_config_version:{proposal_id:374, config_seq:375}, ms_ack_list:0, resend_config_version:{proposal_id:374, config_seq:375}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", register_time_us:1727336095809026, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1727336095809026, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f92b3ff1c70}, [ModeMgr]={palf_id:1, self:"192.168.0.45:2882", applied_mode_meta:{version:1, proposal_id:374, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:374, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:374, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1726163155508348002, v:0}, min_block_id:9357, min_block_min_scn:{val:18446744073709551615, v:3}, base_lsn_for_block_gc:{lsn:627966418944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:374}, log_config_meta:{version:3, proposal_id:374, prev:{version:1, config:{config_version:{proposal_id:373, config_seq:374}, log_sync_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:374, config_seq:375}, log_sync_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:19, prev_lsn:{lsn:629872191817}, prev_mode_pid:374}, log_snapshot_meta:{version:1, base_lsn:{lsn:627966418944}, prev_log_info:{log_id:-1, lsn:{lsn:18446744073709551615}, scn:{val:18446744073709551615, v:3}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:374, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:11440128}, readable_log_tail:{lsn:11440128}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:629912457216}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/observer/store/clog/tenant_1/1/meta", dir_fd:125, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:55664640, block_header_serialize_buf_:0x7f92b3ff6438, flashback_version:0}, log_storage:{log_tail:{lsn:629872191939}, readable_log_tail:{lsn:629872191939}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:9386, palf_id:1}, block_mgr:{log_dir:"/root/observer/store/clog/tenant_1/1/log", dir_fd:128, min_block_id:9357, max_block_id:9387, curr_writable_block_id:9386}, logical_block_size_:67104768, curr_block_writable_size_:40265277, block_header_serialize_buf_:0x7f92b3ff80b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1727336098222784, this:0x7f92b3ff49f0}, [Reconfirm]={palf_id:1, self:"192.168.0.45:2882", state:"FINISHED", new_proposal_id:374, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"192.168.0.45:2882", majority_max_accept_pid:373, majority_max_lsn:{lsn:629872191939}, saved_end_lsn:{lsn:629872191939}, last_submit_prepare_req_time_us:1727336098224520, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f92b3ff4430}) [2024-09-26 16:00:28.320880] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2652) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=72] [PALF_DUMP](palf_id=1, self="192.168.0.45:2882", ack_info_list=[{member:{server:"192.168.0.45:2882", timestamp:1, flag:0}, last_ack_time_us:1727336098526269, last_flushed_end_lsn:{lsn:629872191939}}]) [2024-09-26 16:00:28.320887] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4290) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=6] current log_task status(palf_id=1, self="192.168.0.45:2882", log_id=183389609, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.0.45:2882", timestamp:1, flag:0}, last_ack_time_us:1727336098526269, last_flushed_end_lsn:{lsn:629872191939}}]) [2024-09-26 16:00:28.320898] WDIAG [STORAGE] get_server_region (ob_locality_manager.cpp:399) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4018] fail to get server region(ret=-4018, server="192.168.0.45:2882") [2024-09-26 16:00:28.320901] WDIAG [CLOG] get_server_region (ob_locality_adapter.cpp:61) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4018] get_server_region failed(server="192.168.0.45:2882") [2024-09-26 16:00:28.320903] WDIAG [PALF] update_self_region_ (palf_handle_impl.cpp:408) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4018] get_server_region failed(ret=-4018, this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}) [2024-09-26 16:00:28.320906] WDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2658) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4018] update_region failed(ret=-4018, this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}) [2024-09-26 16:00:28.332688] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:28.335467] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:28.335475] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.338393] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337628293559]}) [2024-09-26 16:00:28.338407] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:28.338417] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337628338404) [2024-09-26 16:00:28.338423] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337628238234, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:28.338429] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:28.338438] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:28.338441] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:28.338768] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:28.338778] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:28.338782] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:28.338785] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:28.338788] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:28.338790] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:28.338793] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:28.338796] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:28.338798] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:28.338800] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:28.338803] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:28.338806] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:28.338812] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=5][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:28.338814] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:28.338821] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:28.338824] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:28.338827] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:28.338830] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:28.338832] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:28.338835] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:28.338839] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:28.338845] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:28.338851] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:28.338853] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:28.338855] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:28.338860] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:28.338864] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:28.338866] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885737-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:28.338871] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:28.338874] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:28.338876] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:28.338879] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337628338686, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:28.338882] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:28.338885] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:28.338914] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:28.338919] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337628338918, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=484, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:28.338926] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:28.338929] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:28.338955] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223357574859, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:28.338964] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:28.338970] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:28.338975] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:28.338979] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:28.338985] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.338988] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.338990] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337628338933) [2024-09-26 16:00:28.340440] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:28.340559] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=50, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:28.341165] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=849339, clean_num=31457) [2024-09-26 16:00:28.365748] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:28.365755] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.376784] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:28.376798] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:28.376805] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:28.376809] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:28.376812] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:28.376815] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:28.376816] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=1][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:28.376819] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:28.376822] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:28.376824] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:28.376826] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:28.376828] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:28.376829] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:28.376831] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:28.376833] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:28.376835] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:28.376841] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:28.376843] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:28.376846] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:28.376847] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:28.376850] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-09-26 16:00:28.376855] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=1][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:28.376860] WDIAG [SERVER] do_process (obmp_query.cpp:771) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=4][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-09-26 16:00:28.376865] WDIAG [SERVER] do_process (obmp_query.cpp:903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, session={this:0x7f92a230bdf8, id:3221487638, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-09-26 16:00:28.376888] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4A-0-0] [lt=17] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:28.383296] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=6] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:28.395980] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:28.395988] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.421385] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=2] LogLoopThread round_cost_time(us)(round_cost_time=111) [2024-09-26 16:00:28.426300] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=53] success to notify_need_writing_throttling True [2024-09-26 16:00:28.426307] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.438994] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:28.439011] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337628438989) [2024-09-26 16:00:28.438999] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337628293559]}) [2024-09-26 16:00:28.439018] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337628338426, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:28.439038] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223357675484, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:28.439045] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:28.439054] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:28.439059] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:28.439063] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:28.439070] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.439073] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.439078] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337628439022) [2024-09-26 16:00:28.443848] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.443848] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=20] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.443853] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.443854] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.443856] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.443859] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.444817] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.444820] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.444820] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.444823] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.444823] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.444824] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.445783] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.445788] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.445790] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.445793] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.445792] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.445813] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=21][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.446774] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.446779] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.446781] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.446822] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.446827] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.446829] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.447735] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.447741] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.447745] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.448703] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.448723] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.448751] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=28][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.448751] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.448757] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.448762] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.449700] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.449705] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.449707] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.449722] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.449725] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.449727] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.450672] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.450676] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.450678] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.450679] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.450680] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.450681] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.451639] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.451656] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:33) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=2] === Run print tenant memory usage task === [2024-09-26 16:00:28.451665] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=26][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.451669] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:28.451676] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:28.451696] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=3] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 413,495,296 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 180,654,080 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 88,080,384 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 12,582,912 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 17,969,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 11,292,672 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 92,430,336 limit= 9,223,372,036,854,775,807 [2024-09-26 16:00:28.451786] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:176) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 180,654,080 used= 173,070,752 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] hold= 33,574,912 used= 33,554,464 count= 1 avg_used= 33,554,464 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 18,894,848 used= 18,874,392 count= 1 avg_used= 18,874,392 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 12,460,032 used= 11,125,608 count= 169 avg_used= 65,832 block_cnt= 169 chunk_cnt= 29 mod=[T]ObSessionDIB [MEMORY] hold= 10,776,576 used= 10,735,904 count= 11 avg_used= 975,991 block_cnt= 11 chunk_cnt= 7 mod=IoControl [MEMORY] hold= 8,585,216 used= 8,566,048 count= 1 avg_used= 8,566,048 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 5,251,072 used= 5,238,784 count= 32 avg_used= 163,712 block_cnt= 32 chunk_cnt= 9 mod=Iteratorget_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6309, mem_used=10398720) [2024-09-26 16:00:28.540563] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:28.540587] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337628293559]}) [2024-09-26 16:00:28.540593] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337628540559) [2024-09-26 16:00:28.540599] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337628439022, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:28.540617] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223357777025, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:28.540624] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:28.540632] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:28.540637] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:28.540640] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:28.540647] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.540649] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.540651] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337628540602) [2024-09-26 16:00:28.543619] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:28.543726] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=55, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:28.544329] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=7, clean_time=0, clean_start_pos=880796, clean_num=31457) [2024-09-26 16:00:28.547426] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:28.547433] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.560062] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:00:28.577477] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:28.577486] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.594550] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:783) [2618][AutoLSLocRpc][T0][YB42C0A8002D-0006230057E85876-0-0] [lt=14] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.0.45:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, location:{server:"192.168.0.45:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:374}}]) [2024-09-26 16:00:28.601119] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.602068] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.602082] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.602089] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.607962] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:28.607968] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.623574] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:28.638234] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:28.638241] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.641140] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337628293559]}) [2024-09-26 16:00:28.641157] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:28.641168] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337628641154) [2024-09-26 16:00:28.641174] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337628540602, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:28.641190] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223357876628, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:28.641196] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:28.641204] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:28.641209] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:28.641212] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:28.641218] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.641221] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.641223] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337628641177) [2024-09-26 16:00:28.641540] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=6] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:28.641566] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=24][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:28.641589] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=22][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:28.641592] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:28.641596] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:28.641599] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:28.641602] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:28.641604] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:28.641619] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=15][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:28.641622] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=3][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:28.641624] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:28.641626] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:28.641629] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:28.641632] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:28.641639] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:28.641642] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:28.641646] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:28.641648] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:28.641651] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, ret=-5019) [2024-09-26 16:00:28.641654] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:28.641659] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=4][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:28.641673] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD1-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:28.641714] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=0] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:28.653842] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:28.668496] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:28.668503] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.694871] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1727337628694870, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000090, add_bytes: 0 [2024-09-26 16:00:28.698774] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:28.698791] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.702672] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=25] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:28.729072] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:28.729081] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.732964] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:28.735093] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [2910][T1_RFLWorker][T1][YB42C0A8002D-0006230055684B24-0-0] [lt=6] ObRemoteFetchWorker is running(thread_index=0) [2024-09-26 16:00:28.741757] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:28.741758] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337628293559]}) [2024-09-26 16:00:28.741768] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337628741755) [2024-09-26 16:00:28.741773] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337628641177, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:28.741789] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223357977264, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:28.741794] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:28.741811] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:28.741817] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:28.741819] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:28.741825] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.741827] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.741829] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337628741775) [2024-09-26 16:00:28.745693] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:28.745806] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=53, refresh_score_time=26, wash_time=1) [2024-09-26 16:00:28.746406] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=5, clean_time=0, clean_start_pos=912253, clean_num=31457) [2024-09-26 16:00:28.759358] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:28.759365] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.760295] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1073) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=7] replay status is not follower(min_unreplayed_lsn={lsn:629872191939}, base_lsn={lsn:627966418944}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:00:28.760333] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=38] get_replay_process success(id={id:1}, replayed_log_size=1905772995, unreplayed_log_size=0) [2024-09-26 16:00:28.760356] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:159) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=13] dump tenant replay process(tenant_id=1, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=1817, last_replayed_log_size(MB)=1817, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-09-26 16:00:28.789602] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:28.789619] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.794614] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:28.794625] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337628794612], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:28.794632] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337628794612], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:28.794636] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:28.794639] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:28.811082] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=8][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337628794612]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:28.811090] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=8][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:28.819876] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:28.819883] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.820953] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.820962] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.821824] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.821835] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.821850] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.821852] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.821859] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.821860] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.821858] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.821864] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.821870] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:28.842374] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337628794612]}) [2024-09-26 16:00:28.842377] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:28.842386] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337628842375) [2024-09-26 16:00:28.842390] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337628741775, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:28.842395] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:28.842404] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:28.842407] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:28.842736] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885738-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:28.842747] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885738-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:28.842775] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885738-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:28.842808] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337628842807, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=407, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:28.842817] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:28.842833] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223358078831, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:28.842847] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:28.842854] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:28.842860] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:28.842863] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:28.842869] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.842871] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.842874] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337628842821) [2024-09-26 16:00:28.850151] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:28.850157] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.856987] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337627856676]) [2024-09-26 16:00:28.856995] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337627856676], now0=[mts=1727337627856676], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:28.857001] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337627855408, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:28.857028] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=26][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:28.857036] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337627855408, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:28.857044] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337627855408, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:28.857060] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:28.857066] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:28.857069] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:28.857072] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:28.857076] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:28.857080] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=24, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:28.857084] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] will sleep(sleep_us=24000, remain_us=4694744, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:28.880920] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:28.880930] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.881289] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:24, local_retry_times:24, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:28.881305] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:28.881308] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:28.881310] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:28.881318] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:28.881369] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:28.881373] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:28.911712] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:28.911744] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.911757] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:28.920496] INFO [CLOG] run1 (ob_garbage_collector.cpp:1338) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=2] Garbage Collector is running(seq_=343, gc_interval=10000000) [2024-09-26 16:00:28.920513] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1473) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] self is leader, skip it(ls->get_ls_id()={id:1}) [2024-09-26 16:00:28.920522] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1431) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=4] gc_check_member_list_ cost time(ret=0, time_us=18) [2024-09-26 16:00:28.920526] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] execute_gc cost time(ret=0, time_us=0) [2024-09-26 16:00:28.920530] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-09-26 16:00:28.920534] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2024-09-26 16:00:28.920538] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] execute_gc cost time(ret=0, time_us=4) [2024-09-26 16:00:28.920539] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=1] ObSafeDestroyHandler start process [2024-09-26 16:00:28.920542] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2024-09-26 16:00:28.920550] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=4] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2024-09-26 16:00:28.933505] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=644848, replace_num=15728) [2024-09-26 16:00:28.933515] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:28.941960] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:28.941984] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.942940] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:28.942948] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:28.942967] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337628942937) [2024-09-26 16:00:28.942973] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337628842393, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:28.942990] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223358178465, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:28.942996] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:28.943003] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:28.943008] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:28.943011] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:28.943016] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.943018] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:28.943020] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337628942975) [2024-09-26 16:00:28.943072] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337628794612]}) [2024-09-26 16:00:28.946964] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:28.947066] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:28.947653] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=943710, clean_num=31457) [2024-09-26 16:00:28.954011] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD2-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-09-26 16:00:28.954023] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD2-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:28.954068] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD2-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:28.954101] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:109) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD2-0-0] [lt=0] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:28.954319] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD2-0-0] [lt=7] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-09-26 16:00:28.954336] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD2-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:28.954357] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:325) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD2-0-0] [lt=0] cal_all_part_disk_default_percentage succ(data_dir="/root/observer/store/sstable", clog_dir="/root/observer/store/clog", shared_mode=true, data_disk_total_size=99765694464, data_disk_default_percentage=60, clog_disk_total_size=99765694464, clog_disk_default_percentage=30) [2024-09-26 16:00:28.954362] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:361) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD2-0-0] [lt=5] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=99765694464, disk_size=15032385536) [2024-09-26 16:00:28.954365] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD2-0-0] [lt=3] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=99765694464) [2024-09-26 16:00:28.954369] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD2-0-0] [lt=3] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=99765694464, this={dir::"/root/observer/store/clog/log_pool", dir_fd:15, meta_fd:16, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:9388, max_block_id:9581, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2024-09-26 16:00:28.954375] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:133) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD2-0-0] [lt=6] refresh tenant config(tenants=[], ret=-5019) [2024-09-26 16:00:28.968430] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:28.972245] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:28.972251] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:28.992920] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=35] new mysql sessid created(conn.sessid_=3221500684, support_ssl=false) [2024-09-26 16:00:28.992942] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221500684, sess.client_addr_="127.0.0.1:36318") [2024-09-26 16:00:28.992947] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f9267204030, session_id:3221500684, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:28.992965] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:28.992973] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:28.993007] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9B-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:28.993173] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9B-0-0] [lt=3][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:28.993180] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9B-0-0] [lt=6][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"l1MpnXJL!Y"838_Uj9|&"}, ret=-4043) [2024-09-26 16:00:28.993185] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9B-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:28.993188] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9B-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:28.993196] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9B-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:28.993207] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 12086.00) [2024-09-26 16:00:28.993215] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9B-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500684, proxy_sessid:0}) [2024-09-26 16:00:28.993219] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9B-0-0] [lt=4] mark session id unused(sessid=3221500684) [2024-09-26 16:00:28.993213] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:28.993223] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9B-0-0] [lt=2][errcode=0] server close connection(sessid=3221500684, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:28.993226] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9B-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500684, proxy_sessid=0) [2024-09-26 16:00:28.993225] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:28.993228] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9B-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:28.993228] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:28.993230] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9B-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500684, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:28.993266] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f9267204030, session_id:3221500684, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337628992826, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:28.993275] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] kill and revert session(conn.sessid_=3221500684, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:28.993277] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=2] can close safely, do destroy(*s={this:0x7f9267204030, session_id:3221500684, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337628992826, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:28.993281] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500684, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:29.002542] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:29.002549] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.021376] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9B-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:29.021389] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9B-0-0] [lt=12][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:29.021437] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9B-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:29.021495] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=0] ls blacklist refresh finish(cost_time=427) [2024-09-26 16:00:29.030034] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F2-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:29.030045] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F2-0-0] [lt=11][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:29.030053] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F2-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:29.030058] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F2-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:29.031099] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:29.031111] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:29.031116] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:29.031119] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:29.031122] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:29.031124] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:29.031127] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:29.031132] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=4][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:29.031134] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:29.031136] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:29.031138] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:29.031140] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:29.031141] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:29.031144] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:29.031149] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:29.031151] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:29.031155] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=3][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:29.031164] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85876-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:29.033799] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:29.033807] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.040635] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=8] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:29.040889] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F42-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:29.040899] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F42-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:29.040933] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F42-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:29.043563] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337628794612]}) [2024-09-26 16:00:29.043554] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:29.043574] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D817-0-0] [lt=6][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337629043556}) [2024-09-26 16:00:29.043581] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337629043551) [2024-09-26 16:00:29.043586] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337628942975, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:29.043604] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223358279062, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:29.043611] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:29.043618] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:29.043623] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:29.043626] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:29.043632] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.043635] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.043645] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337629043590) [2024-09-26 16:00:29.064076] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:29.064084] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.077755] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=14][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:215, tid:2464}]) [2024-09-26 16:00:29.079705] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=10] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:29.094316] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:29.094333] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.124688] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:29.124694] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.134362] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:29.137363] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:221) [2596][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=31] [ASYNC TASK QUEUE](queue_size=0) [2024-09-26 16:00:29.144152] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337628794612]}) [2024-09-26 16:00:29.144150] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:29.144162] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337629144145) [2024-09-26 16:00:29.144167] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337629043589, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:29.144187] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223358379679, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:29.144194] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:29.144202] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:29.144207] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:29.144211] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:29.144235] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.144239] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.144241] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337629144170) [2024-09-26 16:00:29.148185] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:29.148288] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:29.148871] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=975167, clean_num=31457) [2024-09-26 16:00:29.150991] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-09-26 16:00:29.151339] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=37] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-09-26 16:00:29.151351] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2024-09-26 16:00:29.151364] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=12][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2024-09-26 16:00:29.151367] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2024-09-26 16:00:29.151371] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:29.151374] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:29.151377] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2024-09-26 16:00:29.151380] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:29.151381] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:29.151384] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:29.151386] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:29.151388] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:29.151390] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:29.151392] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:29.151399] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:29.151401] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:29.151404] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:29.151410] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:29.151412] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-09-26 16:00:29.151415] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:29.151418] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:29.151424] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:29.151430] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:29.151433] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:29.151434] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:29.151440] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:29.151444] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:29.151447] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD1-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5019, ret=-5019) [2024-09-26 16:00:29.151449] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2024-09-26 16:00:29.151452] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:29.151454] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:29.151457] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337629151264, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2024-09-26 16:00:29.151461] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:29.151466] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:479) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2024-09-26 16:00:29.151496] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:141) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] failed to fetch max rule version(ret=-5019, tenant_id=1) [2024-09-26 16:00:29.151499] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:251) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] failed to fetch max rule version(ret=-5019) [2024-09-26 16:00:29.151501] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:94) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] failed to sync rule from inner table(ret=-5019) [2024-09-26 16:00:29.154902] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:29.154918] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.185180] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:29.185197] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.200059] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:29.200072] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:29.200077] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:29.215461] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:29.215467] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.230421] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:29.230443] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=21][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:29.230449] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:29.230452] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:29.230455] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:29.230457] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:29.230461] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:29.230463] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:29.230465] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:29.230467] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:29.230469] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:29.230471] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:29.230474] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:29.230476] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:29.230482] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:29.230484] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:29.230486] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:29.230490] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:29.230492] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:29.230495] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:29.230498] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:29.230504] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:29.230509] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:29.230511] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:29.230513] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:29.230518] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:29.230522] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:29.230525] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:29.230528] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:29.230531] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:29.230533] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:29.230536] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337629230358, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:29.230539] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:29.230542] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:29.230568] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:29.230573] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=5][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:29.230576] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=3][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:29.230578] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:29.230581] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:29.230583] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:29.230586] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485885-0-0] [lt=3][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:29.244341] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:29.244355] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337629244338) [2024-09-26 16:00:29.244360] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337629144170, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:29.244377] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223358480337, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:29.244382] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:29.244390] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:29.244395] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:29.244398] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:29.244404] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.244406] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.244411] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337629244363) [2024-09-26 16:00:29.244422] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337628794612]}) [2024-09-26 16:00:29.245690] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15] success to notify_need_writing_throttling True [2024-09-26 16:00:29.245703] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.270916] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337629270915, bytes: 14591650, bw: 0.003621 MB/s, add_ts: 1006731, add_bytes: 3822 [2024-09-26 16:00:29.275979] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:29.275995] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.292629] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337630221980, now=[mts=1727337628292341]) [2024-09-26 16:00:29.292641] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=12][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337630221980, now=[mts=1727337628292341], now0=[mts=1727337628292341], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:29.292649] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926918f940, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337628291620, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:29.292668] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=18][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926918f940}) [2024-09-26 16:00:29.292676] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926918f940, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337628291620, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:29.292684] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926918f940, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337628291620, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337628292318, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926918f940}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:29.292702] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:29.292706] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:29.292709] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:29.292711] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:29.292714] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:29.292718] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:29.292722] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=999258, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337630291980) [2024-09-26 16:00:29.292726] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:29.292733] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:29.292734] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=1][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:29.292736] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:29.292741] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:29.292779] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:29.292782] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:29.295638] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.295646] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337629295636], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:29.295652] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337629295636], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:29.295656] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337629295636]}) [2024-09-26 16:00:29.295901] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D3670-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:29.300534] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [2916][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] start to check and clear obj lock(ls->get_ls_id()={id:1}) [2024-09-26 16:00:29.306353] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15] success to notify_need_writing_throttling True [2024-09-26 16:00:29.306361] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.334644] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:29.336464] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:29.336475] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.344382] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:29.344393] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337629344380) [2024-09-26 16:00:29.344398] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337629244363, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:29.344403] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:29.344425] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:29.344428] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:29.344743] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=5] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:29.344776] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=31][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:29.344781] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:29.344784] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:29.344788] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:29.344791] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:29.344794] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:29.344796] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:29.344798] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:29.344800] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:29.344802] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:29.344804] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:29.344807] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:29.344811] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:29.344817] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:29.344822] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:29.344826] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:29.344828] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:29.344831] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:29.344833] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:29.344830] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337629295636]}) [2024-09-26 16:00:29.344835] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:29.344841] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:29.344846] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:29.344849] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:29.344850] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:29.344855] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:29.344859] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:29.344862] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885739-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:29.344865] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:29.344868] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:29.344872] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:29.344875] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337629344669, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:29.344879] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:29.344881] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:29.344911] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:29.344917] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337629344916, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=496, wlock_time=13, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:29.344924] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:29.344927] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:29.344944] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223358581286, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:29.344948] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:29.344955] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:29.344959] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:29.344962] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:29.344967] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.344970] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.344972] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337629344931) [2024-09-26 16:00:29.349391] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:29.349509] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=50, refresh_score_time=13, wash_time=2) [2024-09-26 16:00:29.350110] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=1006624, clean_num=31457) [2024-09-26 16:00:29.357056] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=9][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337629295636]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:29.357068] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=11][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:29.360971] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [2940][T1_ArcFetcher][T1][YB42C0A8002D-0006230055E84B24-0-0] [lt=9] ObArchiveFetcher is running(thread_index=0) [2024-09-26 16:00:29.366839] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27] success to notify_need_writing_throttling True [2024-09-26 16:00:29.366847] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.380721] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=8] current monitor number(seq_=-1) [2024-09-26 16:00:29.383468] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:29.398110] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:29.398119] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.428363] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:29.428371] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.432573] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] ====== tenant freeze timer task ====== [2024-09-26 16:00:29.432586] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:29.432617] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337625619356, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337625619319, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337625619373, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:29.432641] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:29.432661] INFO [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:665) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17] TxData Memory Statistic(logstream info): (ls_id={id:1}, Frozen TxData Memory(MB)=22, Active TxData Memory(MB)=76) [2024-09-26 16:00:29.432667] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:571) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] TxData Memory Statistic : (Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:29.432697] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [2994][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D3-0-0] [lt=4] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:29.432704] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [2994][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D3-0-0] [lt=3] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:29.432709] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [2994][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D3-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:29.432712] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [2994][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D3-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:29.432715] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [2994][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D3-0-0] [lt=2][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:29.432719] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [2994][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D3-0-0] [lt=3][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:29.432723] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [2994][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D3-0-0] [lt=4][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:29.432733] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [2994][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D3-0-0] [lt=10] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:29.432736] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [2994][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D3-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:29.432738] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [2994][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D3-0-0] [lt=1] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:29.432740] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [2994][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D3-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:29.432766] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851D3-0-0] [lt=4] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:29.444603] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.444611] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.444610] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:29.444616] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.444623] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337629444607) [2024-09-26 16:00:29.444628] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337629344401, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:29.444645] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223358681015, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:29.444653] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:29.444660] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:29.444665] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:29.444670] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:29.444675] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.444677] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.444679] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337629444631) [2024-09-26 16:00:29.444684] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.444688] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.444691] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.445925] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.445930] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.445932] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.445938] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337629295636]}) [2024-09-26 16:00:29.445926] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.445948] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.445952] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.446974] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.446979] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.446981] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.446976] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.446998] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.447002] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.447883] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.447888] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.447890] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.447887] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.447904] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.447906] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.448928] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.448938] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.448940] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.448955] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.448958] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.448961] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.449488] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.449493] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.449495] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.449509] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.449511] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.449513] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.450819] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.450821] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.450822] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.450823] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.450825] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.450826] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.451825] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.451839] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.451842] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.451858] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.451863] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.451866] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:29.452770] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.452768] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.453782] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.453781] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.454729] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.454738] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.455709] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.455715] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.456670] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.456685] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.457654] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.457655] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.458635] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.458664] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.458680] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:29.458696] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.459611] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.459623] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.460566] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.460583] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.461231] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.461250] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.462531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.462603] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.463511] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.463548] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.464535] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.464539] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.465522] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.465558] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.466488] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.466497] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.467474] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.467475] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.468457] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.468467] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.469769] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.469791] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.470302] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.470302] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.471348] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.471343] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.472331] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.472332] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.473288] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.473340] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.474267] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.474285] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.475223] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.475246] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.475895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.475909] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.477185] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.477221] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.479152] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.479185] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.480128] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.480152] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=25][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.481149] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.481221] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.482087] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.482173] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.484034] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.484075] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.485091] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.485112] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.485981] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.486018] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.486993] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.487032] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.487585] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.487596] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.488902] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.488920] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.488921] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:29.488936] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.489922] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.489928] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.491837] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.491873] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.493808] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.493863] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.494807] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.494825] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.495767] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.495770] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.497717] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.497720] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:29.519198] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:29.519208] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.522082] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F3-0-0] [lt=30][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:29.522096] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F3-0-0] [lt=13][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:29.522105] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F3-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:29.522111] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F3-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:29.534819] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:29.538692] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=18] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6321, mem_used=10398720) [2024-09-26 16:00:29.544686] WDIAG [PALF] convert_to_ts (scn.cpp:265) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2024-09-26 16:00:29.544695] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1727337629544685, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1727337629544685, force_self_check=false, weak_read_refresh_interval=100000) [2024-09-26 16:00:29.544709] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:29.544717] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337629544707) [2024-09-26 16:00:29.544721] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337629444631, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:29.544738] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223358781036, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:29.544744] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:29.544751] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:29.544756] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:29.544759] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:29.544774] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.544778] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.544780] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337629544724) [2024-09-26 16:00:29.546245] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337629295636]}) [2024-09-26 16:00:29.550514] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:29.550532] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.550656] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:29.550774] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=47, refresh_score_time=32, wash_time=1) [2024-09-26 16:00:29.551423] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1038081, clean_num=31457) [2024-09-26 16:00:29.560227] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=17] gc stale ls task succ [2024-09-26 16:00:29.580729] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:29.580740] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.601864] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.601888] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.601902] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.601909] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.611031] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:29.611039] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.628601] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:29.641290] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:29.641300] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.644727] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get_max_applied_scn(scn={val:1727114270763524000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:374, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:1727114270763524000, v:0}, max_applied_cb_scn_:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:29.644755] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get_min_unreplayed_log_info(lsn={lsn:629872191939}, scn={val:1727114270763524001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:00:29.644777] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=223358881121, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:29.644783] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:29.644790] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:29.644796] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:29.644799] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:29.644806] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.644809] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.644811] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337629644716) [2024-09-26 16:00:29.644820] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:29.646672] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337629295636]}) [2024-09-26 16:00:29.653993] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=7] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:29.671554] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:29.671562] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.701087] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1727337629701086, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006216, add_bytes: 0 [2024-09-26 16:00:29.701905] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:00:29.701914] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.732211] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:29.732222] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.735048] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:29.744828] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:29.744846] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337629744824) [2024-09-26 16:00:29.744853] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337629544724, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:29.744870] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223358981326, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:29.744877] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:29.744885] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:29.744890] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:29.744893] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:29.744900] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.744902] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.744904] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337629744856) [2024-09-26 16:00:29.746772] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337629295636]}) [2024-09-26 16:00:29.751763] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:29.751881] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=53, refresh_score_time=28, wash_time=2) [2024-09-26 16:00:29.752484] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=1069538, clean_num=31457) [2024-09-26 16:00:29.756549] INFO [STORAGE.TRANS] print_statistics (ob_xa_define.cpp:467) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=14] xa statistics(total_active_xa_ctx_count=0, total_standby_clearup_count=0, total_success_xa_start=0, total_failure_xa_start=0, total_success_xa_prepare=0, total_failure_xa_prepare=0, total_success_xa_1pc_commit=0, total_failure_xa_1pc_commit=0, total_success_xa_2pc_commit=0, total_failure_xa_2pc_commit=0, total_failure_xa_rollback=0, total_success_dblink_promotion=0, total_failure_dblink_promotion=0, total_success_dblink=0, total_failure_dblink=0) [2024-09-26 16:00:29.756559] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=9] XA scheduler heartbeat task statistics(avg_time=11) [2024-09-26 16:00:29.762396] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:29.762404] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.778968] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=4] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-09-26 16:00:29.778977] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-09-26 16:00:29.778982] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=4] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=686) [2024-09-26 16:00:29.792680] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:00:29.792688] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.795986] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:29.795999] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337629795984], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:29.796006] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337629795984], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:29.796011] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:29.796015] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:29.821001] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.821052] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.821060] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.821065] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.821073] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.821974] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.821976] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.821985] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.821994] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.822000] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.822006] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:29.822963] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24] success to notify_need_writing_throttling True [2024-09-26 16:00:29.822971] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.845433] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:29.845443] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337629845429) [2024-09-26 16:00:29.845461] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337629744856, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:29.845467] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:29.845475] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:29.845479] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:29.845802] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:29.845820] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=17][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:29.845824] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:29.845828] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:29.845831] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:29.845833] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:29.845837] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:29.845839] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:29.845841] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:29.845843] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:29.845845] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:29.845847] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:29.845850] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:29.845852] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:29.845858] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:29.845860] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:29.845863] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:29.845865] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:29.845867] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:29.845870] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:29.845872] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:29.845877] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:29.845883] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:29.845885] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:29.845886] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:29.845892] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:29.845895] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:29.845897] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573A-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:29.845899] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:29.845902] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:29.845904] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:29.845907] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337629845725, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:29.845910] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:29.845912] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:29.845940] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:29.845943] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337629845943, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=471, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:29.845950] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:29.845952] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:29.845968] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223359081889, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:29.845972] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:29.845978] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:29.845982] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:29.845984] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:29.845989] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.845991] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.845992] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337629845956) [2024-09-26 16:00:29.847161] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337629795984]}) [2024-09-26 16:00:29.853263] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:29.853270] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.859435] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, table_name.ptr()="data_size:16, data:5F5F616C6C5F6D657267655F696E666F", ret=-5019) [2024-09-26 16:00:29.859447] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, ret=-5019) [2024-09-26 16:00:29.859450] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_merge_info, db_name=oceanbase) [2024-09-26 16:00:29.859454] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_merge_info) [2024-09-26 16:00:29.859457] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:29.859458] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:29.859461] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_merge_info' doesn't exist [2024-09-26 16:00:29.859463] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:29.859465] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:29.859466] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:29.859468] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:29.859470] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:29.859472] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:29.859474] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:29.859480] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:29.859483] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:29.859485] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:29.859489] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:29.859494] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_merge_info WHERE tenant_id = '1', ret=-5019) [2024-09-26 16:00:29.859497] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:29.859526] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=28][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:29.859533] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:29.859538] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:29.859540] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:29.859541] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:29.859546] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:29.859550] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:29.859553] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507E-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, aret=-5019, ret=-5019) [2024-09-26 16:00:29.859555] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:29.859558] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:29.859561] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:29.859566] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337629859391, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:29.859569] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:29.859571] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:49) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:29.859593] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:890) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] failed to load global merge info(ret=-5019, ret="OB_TABLE_NOT_EXIST", global_merge_info={tenant_id:1, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2024-09-26 16:00:29.859606] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1005) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] fail to refresh merge info(tmp_ret=-5019, tmp_ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:29.859808] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:29.859815] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:29.881653] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337628881368]) [2024-09-26 16:00:29.881669] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337628881368], now0=[mts=1727337628881368], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:29.881677] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337628880849, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:29.881695] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:29.881705] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337628880849, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:29.881713] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337628880849, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:29.881732] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=18][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:29.881737] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:29.881740] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:29.881743] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:29.881745] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:29.881748] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=25, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:29.881753] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] will sleep(sleep_us=25000, remain_us=3670075, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:29.883522] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:29.883532] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.906844] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:25, local_retry_times:25, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:29.906866] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:29.906871] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:29.906872] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:29.906884] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:29.906929] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:29.906933] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:29.913742] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:29.913752] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.913786] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:29.915761] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3] ====== checkpoint timer task ====== [2024-09-26 16:00:29.916294] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x7f92a6a04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1726077600526835000, v:0}, last_push_gc_task_ts_:1727334198745636, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:29.916309] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] tx ctx memtable get rec scn(this={ObITable:{this:0x7f92a22a4080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f92a22a4080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:29.916325] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1723174350926847000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-09-26 16:00:29.916348] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:29.916379] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=28] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}, min_rec_scn={val:1726163586617842000, v:0}, ls_id_={id:1}) [2024-09-26 16:00:29.916395] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:29.916399] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=4] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:29.916402] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:29.916409] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1726163586617842000, v:0}, checkpoint_scn_in_ls_meta={val:1726163586617842000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-09-26 16:00:29.916418] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}, curr_base_lsn={lsn:627966418944}, new_base_lsn={lsn:627966418944}, lsn={lsn:627966418944}) [2024-09-26 16:00:29.916426] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:627966418944}, ls->get_ls_id()={id:1}) [2024-09-26 16:00:29.916430] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-09-26 16:00:29.935349] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:29.944076] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:29.944084] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.946100] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:29.946108] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:29.946129] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337629946097) [2024-09-26 16:00:29.946135] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337629845464, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:29.946152] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223359182505, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:29.946158] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:29.946165] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:29.946169] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:29.946171] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:29.946177] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.946179] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:29.946181] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337629946138) [2024-09-26 16:00:29.948965] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337629795984]}) [2024-09-26 16:00:29.952959] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:29.953077] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:29.953700] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1100995, clean_num=31457) [2024-09-26 16:00:29.966523] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [2708][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=206] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"2GB", log_disk_size:"2GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, mode:0, create_timestamp:1710828706488537, is_removed:false}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=15, log_id=1828900, offset=14449774}, ls_meta_entry:[10007](ver=0,mode=0,seq=609892), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:2}, create_status:1}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:49, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:79460263, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:1720, recv_lp_rpc_cnt:0, recv_mysql_cnt:1141, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:9679698, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:group_id = 2,queue_size = 0,recv_req_cnt = 7464,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334211521784 group_id = 10,queue_size = 0,recv_req_cnt = 3412,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334216415596 group_id = 9,queue_size = 0,recv_req_cnt = 7056,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334199387293 group_id = 1,queue_size = 0,recv_req_cnt = 4466,min_worker_cnt = 8,max_worker_cnt = 8,worker_cnt = 0,token_change = 1727336105580026 , rpc_stat_info: pcode=0x150a:cnt=20 pcode=0x150b:cnt=20 pcode=0x4a9:cnt=10 pcode=0x113:cnt=5, token_change_ts:1727334199363389, tenant_role:1}) [2024-09-26 16:00:29.966712] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [2708][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=188] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1727334198122806, is_removed:false}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:2}, create_status:1}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:12, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:2133720, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3226307, workers:12, nesting workers:0, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1727334198125585, tenant_role:0}) [2024-09-26 16:00:29.970471] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:29.974369] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:29.974375] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:29.976331] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-09-26 16:00:29.976341] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=9] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:29.976355] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=13] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:29.976359] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=3] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:29.978033] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] Dumping All Cache Objs(alloc_obj_list.count()=3, alloc_obj_list=[{obj_id:78, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157779}, {obj_id:79, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:5, added_to_lc:true, mem_used:175172}]) [2024-09-26 16:00:29.978046] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=12] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:29.994047] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500691, support_ssl=false) [2024-09-26 16:00:29.994070] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221500691, sess.client_addr_="127.0.0.1:36320") [2024-09-26 16:00:29.994075] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f9264004030, session_id:3221500691, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:29.994093] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:29.994102] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=2] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:29.994151] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9A-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:29.994326] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9A-0-0] [lt=6][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:29.994335] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9A-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"lg@G$~n0A70*!gb:7D2l"}, ret=-4043) [2024-09-26 16:00:29.994355] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9A-0-0] [lt=18][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:29.994359] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9A-0-0] [lt=4][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:29.994367] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9A-0-0] [lt=3] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:29.994404] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9A-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500691, proxy_sessid:0}) [2024-09-26 16:00:29.994409] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9A-0-0] [lt=5] mark session id unused(sessid=3221500691) [2024-09-26 16:00:29.994413] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9A-0-0] [lt=2][errcode=0] server close connection(sessid=3221500691, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:29.994417] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9A-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500691, proxy_sessid=0) [2024-09-26 16:00:29.994419] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9A-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:29.994425] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9A-0-0] [lt=5] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500691, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:29.994466] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f9264004030, session_id:3221500691, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337629994057, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:29.994483] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] kill and revert session(conn.sessid_=3221500691, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:29.994486] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9264004030, session_id:3221500691, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337629994057, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:29.994491] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500691, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:29.998776] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10755.00) [2024-09-26 16:00:29.998784] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:29.998795] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:29.998799] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:30.004572] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:30.004578] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.027734] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F4-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:30.027745] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F4-0-0] [lt=11][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:30.027756] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F4-0-0] [lt=11][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:30.027761] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F4-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:30.032260] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:30.032273] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:30.032278] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:30.032281] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:30.032300] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=17][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:30.032302] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:30.032305] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:30.032308] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:30.032310] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:30.032312] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:30.032323] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=11][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:30.032326] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.032367] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85877-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:30.034921] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:30.034926] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.041754] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=6] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:30.042015] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F44-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:30.042024] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F44-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.042044] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F44-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:30.046645] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:30.046655] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337630046643) [2024-09-26 16:00:30.046660] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337629946137, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:30.046676] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223359282135, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.046681] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:30.046688] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:30.046693] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:30.046696] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:30.046701] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.046703] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.046705] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337630046662) [2024-09-26 16:00:30.049564] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337629795984]}) [2024-09-26 16:00:30.049572] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D823-0-0] [lt=8][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337630049554}) [2024-09-26 16:00:30.064975] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14] success to notify_need_writing_throttling True [2024-09-26 16:00:30.064980] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.080804] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=8] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:30.095508] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:30.095517] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.125770] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:30.125780] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.135507] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:30.147211] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:30.147222] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337630147208) [2024-09-26 16:00:30.147227] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337630046662, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:30.147243] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223359383275, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.147248] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:30.147255] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:30.147260] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:30.147262] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:30.147267] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.147269] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.147271] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337630147230) [2024-09-26 16:00:30.150143] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337629795984]}) [2024-09-26 16:00:30.154217] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:30.154333] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=50, refresh_score_time=27, wash_time=1) [2024-09-26 16:00:30.154946] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=1132452, clean_num=31457) [2024-09-26 16:00:30.156001] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25] success to notify_need_writing_throttling True [2024-09-26 16:00:30.156009] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.184573] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:62, tid:2894}]) [2024-09-26 16:00:30.186240] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:30.186250] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.197044] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:00:30.197054] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=7] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:30.197063] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:30.216560] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:30.216569] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.231526] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:30.231537] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:30.231542] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:30.231545] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:30.231548] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:30.231551] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:30.231554] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:30.231556] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:30.231558] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:30.231560] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:30.231562] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:30.231564] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:30.231567] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:30.231569] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:30.231575] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:30.231577] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:30.231580] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:30.231582] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:30.231585] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:30.231587] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:30.231590] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:30.231595] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:30.231601] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:30.231603] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:30.231605] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:30.231611] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:30.231614] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:30.231616] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:30.231619] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:30.231621] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=1][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:30.231623] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:30.231626] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337630231435, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:30.231629] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:30.231632] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:30.231660] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:30.231662] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:30.231665] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:30.231666] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=1][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:30.231669] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:30.231672] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:30.231674] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485886-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:30.246613] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:00:30.246623] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.247822] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:30.247833] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337630247819) [2024-09-26 16:00:30.247838] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337630147230, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:30.247853] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223359483700, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.247859] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:30.247866] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:30.247870] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:30.247873] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:30.247878] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.247880] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.247882] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337630247841) [2024-09-26 16:00:30.250391] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337629795984]}) [2024-09-26 16:00:30.277021] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:30.277030] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.279080] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO [ratelimit] time: 1727337630279079, bytes: 14595190, bw: 0.003349 MB/s, add_ts: 1008164, add_bytes: 3540 [2024-09-26 16:00:30.292964] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337630221980, now=[mts=1727337629292778]) [2024-09-26 16:00:30.292973] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337630221980, now=[mts=1727337629292778], now0=[mts=1727337629292778], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:30.292979] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926918f940, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337629291777, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:30.292997] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926918f940}) [2024-09-26 16:00:30.293005] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926918f940, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337629291777, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:30.293013] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926918f940, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337629291777, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337628292318, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926918f940}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:30.293043] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=29][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:30.293047] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:30.293050] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:30.293053] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:30.293055] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:30.293059] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:30.293064] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:30.293071] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:30.293073] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:30.293075] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:30.293080] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:30.293084] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=3] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000766) [2024-09-26 16:00:30.293087] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:30.293092] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:30.293095] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:30.293098] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:30.293100] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=1][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:30.293105] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:30.293131] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:30.293134] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:30.293136] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:30.293139] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:30.293143] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=4][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:30.293146] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=3][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:30.293254] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C77-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:30.293263] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:30.293265] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:30.293276] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:30.293285] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=8][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:30.293591] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:00:30.293599] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=8][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:30.295814] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D36E6-0-0] [lt=11][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:30.296336] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.296345] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337630296333], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:30.296364] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337630296333], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:30.307436] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:30.307446] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.335747] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:30.337702] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:00:30.337727] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.348417] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:30.348430] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337630348414) [2024-09-26 16:00:30.348436] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337630247841, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:30.348442] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:30.348450] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:30.348453] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:30.348808] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:30.348819] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:30.348823] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:30.348839] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:30.348843] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:30.348845] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:30.348849] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:30.348852] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:30.348854] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:30.348856] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:30.348858] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:30.348860] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:30.348864] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:30.348866] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:30.348873] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:30.348876] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:30.348879] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:30.348882] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:30.348885] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:30.348888] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:30.348893] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:30.348898] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:30.348903] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:30.348906] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:30.348907] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:30.348912] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:30.348917] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=4][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:30.348920] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573B-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:30.348929] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:30.348932] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:30.348937] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:30.348940] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337630348737, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:30.348943] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:30.348946] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:30.348973] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:30.348977] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337630348976, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=530, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:30.348984] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:30.348987] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:30.349003] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223359584958, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.349008] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:30.349014] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:30.349018] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:30.349020] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:30.349025] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.349028] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.349030] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337630348991) [2024-09-26 16:00:30.350393] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337630296333]}) [2024-09-26 16:00:30.355412] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:30.355538] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=52, refresh_score_time=23, wash_time=2) [2024-09-26 16:00:30.356139] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1163909, clean_num=31457) [2024-09-26 16:00:30.368957] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:30.368963] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.380759] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=6] current monitor number(seq_=-1) [2024-09-26 16:00:30.383594] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:30.399206] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:30.399213] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.399204] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [2925][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=13] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f92b77d02b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-09-26 16:00:30.429498] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:30.429507] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.446091] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.446111] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.446112] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.446120] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.446119] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.446124] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.446125] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.446127] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.446129] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.447066] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.447068] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.447071] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.447074] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.447079] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.447081] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.447082] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.447083] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.447085] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.448014] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.448024] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.448029] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.448086] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.448090] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.448092] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.448094] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.448096] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.448099] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.449009] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.449018] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.449023] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.449027] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.449029] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.449031] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.449029] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:30.449036] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.449038] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.449039] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.449056] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337630449026) [2024-09-26 16:00:30.449062] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337630348439, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:30.449081] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223359685544, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.449087] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:30.449095] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:30.449100] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:30.449102] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:30.449109] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.449111] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.449113] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337630449066) [2024-09-26 16:00:30.449992] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.449998] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.450003] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.450008] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.450012] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.450014] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.450018] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.450019] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.450021] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.451996] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.452000] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.452002] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:30.452009] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337630296333]}) [2024-09-26 16:00:30.452017] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.452025] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.452937] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.452984] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.452990] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.453899] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.453906] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.453908] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.454895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.454898] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.454902] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.455885] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.455899] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.455903] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.456844] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.456850] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.456896] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.457761] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.457767] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.457799] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.458790] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.458807] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.458813] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.459308] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.459330] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.459380] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.459721] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:30.459726] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.459824] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.459889] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.459906] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.460426] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.460427] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.460483] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.461098] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.461108] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.461913] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.462139] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.462166] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.462493] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.462984] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=25][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.462994] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.463052] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.463655] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.463997] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.464018] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.464254] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.464574] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.464594] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.464832] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.465561] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.465570] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.465798] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.466406] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.466418] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.466482] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.467242] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.467253] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.467595] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.467763] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.467944] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.468277] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.468279] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.468558] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.468845] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.468845] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.469137] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.469388] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.469480] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.469708] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.469998] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.470050] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.470253] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.470627] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.471347] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.471974] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.472649] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.473213] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.474401] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.475076] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.475088] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.475144] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=24][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.475625] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.475665] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.475716] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.476225] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:30.490041] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:30.490053] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.520302] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:30.520315] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.525387] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F5-0-0] [lt=48][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:30.525400] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F5-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:30.525409] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F5-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:30.525414] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F5-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:30.535931] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:30.538906] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=1, size_used=6329, mem_used=10398720) [2024-09-26 16:00:30.549641] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:30.549651] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337630549638) [2024-09-26 16:00:30.549656] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337630449066, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:30.549674] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223359785234, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.549680] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:30.549688] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:30.549693] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:30.549695] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:30.549701] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.549703] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.549704] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337630549659) [2024-09-26 16:00:30.551587] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:30.551592] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.552555] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337630296333]}) [2024-09-26 16:00:30.556594] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:30.556702] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=61, refresh_score_time=16, wash_time=2) [2024-09-26 16:00:30.557315] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=1195366, clean_num=31457) [2024-09-26 16:00:30.560389] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:00:30.581871] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:30.581879] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.601509] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.601521] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.601526] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.602378] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.611913] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:30.611921] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.620579] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] ====== traversal_flush timer task ====== [2024-09-26 16:00:30.620621] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9272926030, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6163825430799493735, dag_cnt=1, dag_type_cnts=1) [2024-09-26 16:00:30.620629] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] success to create and add dag(ret=0, dag=0x7f9272926030) [2024-09-26 16:00:30.620632] INFO [STORAGE] flush (ob_tx_data_memtable.cpp:917) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] [TX DATA MERGE]schedule flush tx data memtable task done(ret=0, ret="OB_SUCCESS", get_ls_id()={id:1}, this=0x7f92a2294080, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:4, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337630620632, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:30.620656] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1961) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(dag_cnt=1, map_size=1) [2024-09-26 16:00:30.620665] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1971) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(running_dag_net_map_size=0, blocking_dag_net_list_size=0) [2024-09-26 16:00:30.620667] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_COMPACTION_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=1, waiting_dag_count=0) [2024-09-26 16:00:30.620672] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_HA_HIGH", low_limit=8, up_limit=8, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:30.620674] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_COMPACTION_MID", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:30.620676] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_HA_MID", low_limit=5, up_limit=5, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:30.620680] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(priority="PRIO_COMPACTION_LOW", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:30.620682] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_HA_LOW", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:30.620683] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(priority="PRIO_DDL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:30.620685] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_DDL_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:30.620686] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(priority="PRIO_TTL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:30.620688] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:30.620692] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=182) [2024-09-26 16:00:30.620694] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:30.620696] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:00:30.620698] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:30.620699] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:00:30.620679] INFO [MDS] flush (mds_table_mgr.cpp:172) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=29] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:30.620701] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=1) [2024-09-26 16:00:30.620703] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=2) [2024-09-26 16:00:30.620707] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:30.620709] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:00:30.620710] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:30.620712] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:00:30.620713] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:00:30.620715] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:00:30.620716] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:00:30.620716] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=22] add dag success(dag=0x7f9272926560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6659188001345271846, dag_cnt=2, dag_type_cnts=1) [2024-09-26 16:00:30.620718] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:00:30.620719] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9272926560) [2024-09-26 16:00:30.620719] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:00:30.620721] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:00:30.620723] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:00:30.620725] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:00:30.620727] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620723] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, create_flush_dag_time:1727337630620722, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452867, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200240}, checkpoint_unit_ptr:0x7f92a1a70850}, is_freeze_clock:true}) [2024-09-26 16:00:30.620731] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620733] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620734] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620736] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620737] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620739] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620740] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620742] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620730] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337630620709, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.620743] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620745] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620746] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620747] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=14] add dag success(dag=0x7f9272926a90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-9069613569758001279, dag_cnt=3, dag_type_cnts=2) [2024-09-26 16:00:30.620748] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620749] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272926a90) [2024-09-26 16:00:30.620750] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620751] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620751] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, create_flush_dag_time:1727337630620750, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452892, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200489}, checkpoint_unit_ptr:0x7f92a1a70080}, is_freeze_clock:true}) [2024-09-26 16:00:30.620755] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620757] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620758] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620759] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620754] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337630620744, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.620761] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620762] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620763] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272926fc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1288986994123435108, dag_cnt=4, dag_type_cnts=3) [2024-09-26 16:00:30.620764] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620765] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272926fc0) [2024-09-26 16:00:30.620765] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620767] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620768] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620767] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337630620767, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452911, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847222}, checkpoint_unit_ptr:0x7f92a1a71020}, is_freeze_clock:true}) [2024-09-26 16:00:30.620770] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620772] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620773] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620770] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337630620761, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.620777] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:30.620779] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620779] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f92729274f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5355160120942094702, dag_cnt=5, dag_type_cnts=4) [2024-09-26 16:00:30.620780] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-09-26 16:00:30.620781] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f92729274f0) [2024-09-26 16:00:30.620782] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-09-26 16:00:30.620783] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-09-26 16:00:30.620782] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337630620782, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452929, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847223}, checkpoint_unit_ptr:0x7f92a1a717f0}, is_freeze_clock:true}) [2024-09-26 16:00:30.620785] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-09-26 16:00:30.620786] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:30.620788] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:30.620790] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:30.620785] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337630620777, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.620792] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:30.620793] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:30.620795] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:30.620797] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:30.620798] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:30.620794] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405042) [2024-09-26 16:00:30.620802] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:30.620804] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:30.620805] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:30.620806] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:30.620807] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:30.620808] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:30.620810] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:30.620811] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:30.620813] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:30.620809] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405070) [2024-09-26 16:00:30.620814] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:30.620815] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:30.620816] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:30.620817] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:30.620818] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620821] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:30.620822] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620819] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405087) [2024-09-26 16:00:30.620824] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:30.620825] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:30.620826] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620827] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:30.620829] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620830] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:30.620832] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620828] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405104) [2024-09-26 16:00:30.620834] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:30.620834] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:30.620835] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620837] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:30.620838] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] add dag success(dag=0x7f9272927a20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1994541446894784994, dag_cnt=6, dag_type_cnts=5) [2024-09-26 16:00:30.620838] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620839] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272927a20) [2024-09-26 16:00:30.620840] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:30.620841] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620841] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, create_flush_dag_time:1727337630620841, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209556884, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:226}, checkpoint_unit_ptr:0x7f92a1a73f00}, is_freeze_clock:true}) [2024-09-26 16:00:30.620845] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:30.620847] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620848] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:30.620845] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337630620836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.620850] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:30.620851] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0) [2024-09-26 16:00:30.620853] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, scheduled_task_count=0) [2024-09-26 16:00:30.620864] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0) [2024-09-26 16:00:30.620866] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, scheduled_task_count=0) [2024-09-26 16:00:30.620865] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272927f50, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8791616776033068452, dag_cnt=7, dag_type_cnts=6) [2024-09-26 16:00:30.620868] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9272927f50) [2024-09-26 16:00:30.620869] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0) [2024-09-26 16:00:30.620870] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-09-26 16:00:30.620870] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337630620870, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405141, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847408}, checkpoint_unit_ptr:0x7f92a1a746d0}, is_freeze_clock:true}) [2024-09-26 16:00:30.620874] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0) [2024-09-26 16:00:30.620876] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-09-26 16:00:30.620878] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0) [2024-09-26 16:00:30.620879] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, scheduled_task_count=0) [2024-09-26 16:00:30.620873] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337630620851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.620881] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_net_count=0) [2024-09-26 16:00:30.620882] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_net_count=0) [2024-09-26 16:00:30.620883] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_net_count=0) [2024-09-26 16:00:30.620884] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9272928480, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6550230011160095889, dag_cnt=8, dag_type_cnts=7) [2024-09-26 16:00:30.620885] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_net_count=0) [2024-09-26 16:00:30.620886] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272928480) [2024-09-26 16:00:30.620886] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_net_count=0) [2024-09-26 16:00:30.620887] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_net_count=0) [2024-09-26 16:00:30.620888] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_net_count=0) [2024-09-26 16:00:30.620887] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337630620887, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405158, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847409}, checkpoint_unit_ptr:0x7f92a1a75670}, is_freeze_clock:true}) [2024-09-26 16:00:30.620891] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_net_count=0) [2024-09-26 16:00:30.620892] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1991) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=184) [2024-09-26 16:00:30.620890] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337630620881, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.620900] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337630620897, task_id:YB42C0A8002D-0006230056D8C5E8-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.620906] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=5] schedule one task(task=0x7f92729263f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.620915] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337630620914, task_id:YB42C0A8002D-0006230056D8C5E9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.620919] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=3] schedule one task(task=0x7f9272926920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.620924] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630620923, task_id:YB42C0A8002D-0006230056D8C5EA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.620926] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=2] schedule one task(task=0x7f9272926e50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.620929] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630620929, task_id:YB42C0A8002D-0006230056D8C5EB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.620932] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=2] schedule one task(task=0x7f9272927380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.620936] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337630620936, task_id:YB42C0A8002D-0006230056D8C5EC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.620939] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=2] schedule one task(task=0x7f92729278b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.620944] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337630620943, task_id:YB42C0A8002D-0006230056D8C5ED-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.620946] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2] schedule one task(task=0x7f9272927de0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.620959] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.620967] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.620970] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726157244792838000) [2024-09-26 16:00:30.620973] INFO [STORAGE] ready_for_flush (ob_tx_data_memtable.cpp:865) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=2] memtable is frozen yet.(this=0x7f92a2294080) [2024-09-26 16:00:30.620973] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.620976] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.620979] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:30.620985] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.621011] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.621014] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.621016] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:30.621019] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.621042] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.621044] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.621046] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:30.621048] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.620991] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726164780922097000, old_tablet_snaphsot_version=1726164781026522000, tablet={this:0x7f92b863df20, is_inited:true, wash_score:1727337630620660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:487, tablet_addr:[816017-1849-0-2305843626616512512-2](offset=1269760,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, data_tablet_id:{id:200240}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726164781225658000, v:0}, clog_checkpoint_scn:{val:1726164781225658000, v:0}, ddl_checkpoint_scn:{val:1711609700160391001, v:0}, snapshot_version:1726164781026522000, multi_version_start:1726164780922097000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726164780922097000, cur_report_version:1726164780922097000, data_checksum:2501178096, row_count:147053}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:1711609700101236001, v:0}, ddl_snapshot_version:1711609700074566000, max_sync_storage_schema_version:1711609699705840, max_serialized_medium_scn:0, ddl_execution_id:0, ddl_data_format_version:17180000769, ddl_commit_scn:{val:1711609700160391001, v:0}, mds_checkpoint_scn:{val:1726201924319685000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1711609699705840}, table_store_addr:{ptr:{this:0x7f92b863e5d8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844241870487552-0](offset=1179648,size=1148,type=2,seq=0), type:"MAJOR", tablet_id:{id:1152921504606847223}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726164780922097000, v:0}}, snapshot_version:1726164780922097000, max_merge_version:1726164780922097000}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337630620777, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843251880595456-2](offset=1249280,size=226,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276217058-0](offset=1249506,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255101821304-0](offset=1249656,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:1726164780922097000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1711609699719192000, v:0}, create_commit_version:1711609699719192000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a717f0, 0x7f92a1a72f60, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.621054] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:30.621065] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.621065] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.621069] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.621071] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:30.621073] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.621076] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847222}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:30.621086] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.621088] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.621091] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.621093] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.621096] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.621098] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:00:30.621102] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241777443612000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.621410] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f92729289b0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6809498982591245614, dag_cnt=9, dag_type_cnts=8) [2024-09-26 16:00:30.621419] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] success to create and add dag(ret=0, dag=0x7f92729289b0) [2024-09-26 16:00:30.621424] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, create_flush_dag_time:1727337630621424, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405174, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:200004}, checkpoint_unit_ptr:0x7f92a1a74ea0}, is_freeze_clock:true}) [2024-09-26 16:00:30.621432] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337630620897, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621451] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=13] add dag success(dag=0x7f9272928ee0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6815944285528637599, dag_cnt=10, dag_type_cnts=9) [2024-09-26 16:00:30.621471] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=20] success to create and add dag(ret=0, dag=0x7f9272928ee0) [2024-09-26 16:00:30.621474] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, create_flush_dag_time:1727337630621474, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209506214, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1}, checkpoint_unit_ptr:0x7f92a1a76610}, is_freeze_clock:true}) [2024-09-26 16:00:30.621478] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337630621445, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621491] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9272929410, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-4182959058374342994, dag_cnt=11, dag_type_cnts=10) [2024-09-26 16:00:30.621493] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272929410) [2024-09-26 16:00:30.621495] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, create_flush_dag_time:1727337630621494, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405204, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:412}, checkpoint_unit_ptr:0x7f92a1a75e40}, is_freeze_clock:true}) [2024-09-26 16:00:30.621498] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337630621487, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621510] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272929940, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1586518267935340, dag_cnt=12, dag_type_cnts=11) [2024-09-26 16:00:30.621513] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272929940) [2024-09-26 16:00:30.621514] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, create_flush_dag_time:1727337630621514, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405220, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:444}, checkpoint_unit_ptr:0x7f92a1a775b0}, is_freeze_clock:true}) [2024-09-26 16:00:30.621518] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337630621505, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621528] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272929e70, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3044367941103776476, dag_cnt=13, dag_type_cnts=12) [2024-09-26 16:00:30.621531] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272929e70) [2024-09-26 16:00:30.621532] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, create_flush_dag_time:1727337630621532, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455964, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:309}, checkpoint_unit_ptr:0x7f92a1a76de0}, is_freeze_clock:true}) [2024-09-26 16:00:30.621536] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337630621525, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621545] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292a3a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1596830851104282047, dag_cnt=14, dag_type_cnts=13) [2024-09-26 16:00:30.621548] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292a3a0) [2024-09-26 16:00:30.621549] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, create_flush_dag_time:1727337630621549, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455986, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:331}, checkpoint_unit_ptr:0x7f92a1a77d80}, is_freeze_clock:true}) [2024-09-26 16:00:30.621553] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337630621542, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621562] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292a8d0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2360195962081676673, dag_cnt=15, dag_type_cnts=14) [2024-09-26 16:00:30.621565] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292a8d0) [2024-09-26 16:00:30.621566] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, create_flush_dag_time:1727337630621566, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405262, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:458}, checkpoint_unit_ptr:0x7f92a1a7a490}, is_freeze_clock:true}) [2024-09-26 16:00:30.621570] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337630621559, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621580] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292ae00, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5675615614797020460, dag_cnt=16, dag_type_cnts=15) [2024-09-26 16:00:30.621582] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292ae00) [2024-09-26 16:00:30.621584] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, create_flush_dag_time:1727337630621584, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405279, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:487}, checkpoint_unit_ptr:0x7f92a1a79cc0}, is_freeze_clock:true}) [2024-09-26 16:00:30.621587] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337630621576, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621597] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927292b330, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-612435342908088006, dag_cnt=17, dag_type_cnts=16) [2024-09-26 16:00:30.621599] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292b330) [2024-09-26 16:00:30.621601] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, create_flush_dag_time:1727337630621601, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405296, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:455}, checkpoint_unit_ptr:0x7f92a1a794f0}, is_freeze_clock:true}) [2024-09-26 16:00:30.621604] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337630621594, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621614] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927292b860, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7868818205990873505, dag_cnt=18, dag_type_cnts=17) [2024-09-26 16:00:30.621616] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292b860) [2024-09-26 16:00:30.621618] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, create_flush_dag_time:1727337630621618, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405313, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:492}, checkpoint_unit_ptr:0x7f92a1a78550}, is_freeze_clock:true}) [2024-09-26 16:00:30.621621] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337630621611, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621630] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927292bd90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7071731214611448811, dag_cnt=19, dag_type_cnts=18) [2024-09-26 16:00:30.621633] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292bd90) [2024-09-26 16:00:30.621637] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, create_flush_dag_time:1727337630621637, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405347, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:501}, checkpoint_unit_ptr:0x7f92a1a78d20}, is_freeze_clock:true}) [2024-09-26 16:00:30.621641] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337630621628, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621664] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=18] add dag success(dag=0x7f927292c2c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8731750006000903454, dag_cnt=20, dag_type_cnts=19) [2024-09-26 16:00:30.621667] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292c2c0) [2024-09-26 16:00:30.621669] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, create_flush_dag_time:1727337630621669, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405365, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:472}, checkpoint_unit_ptr:0x7f92a1a7ac60}, is_freeze_clock:true}) [2024-09-26 16:00:30.621673] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337630621659, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621683] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927292c7f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6766388540859538791, dag_cnt=21, dag_type_cnts=20) [2024-09-26 16:00:30.621687] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f927292c7f0) [2024-09-26 16:00:30.621689] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, create_flush_dag_time:1727337630621689, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405384, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:342}, checkpoint_unit_ptr:0x7f92a1a7cba0}, is_freeze_clock:true}) [2024-09-26 16:00:30.621692] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337630621680, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621703] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f927292cd20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2986199386776543176, dag_cnt=22, dag_type_cnts=21) [2024-09-26 16:00:30.621706] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292cd20) [2024-09-26 16:00:30.621707] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337630621707, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405418, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:140}, checkpoint_unit_ptr:0x7f92a1a7b430}, is_freeze_clock:true}) [2024-09-26 16:00:30.621711] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337630621700, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621720] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292d250, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6947063304478012270, dag_cnt=23, dag_type_cnts=22) [2024-09-26 16:00:30.621722] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292d250) [2024-09-26 16:00:30.621724] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337630621724, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405436, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101009}, checkpoint_unit_ptr:0x7f92a1a7bc00}, is_freeze_clock:true}) [2024-09-26 16:00:30.621727] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337630621717, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621737] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927292d780, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5843378789243662202, dag_cnt=24, dag_type_cnts=23) [2024-09-26 16:00:30.621739] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292d780) [2024-09-26 16:00:30.621741] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337630621741, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405453, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101010}, checkpoint_unit_ptr:0x7f92a1a7c3d0}, is_freeze_clock:true}) [2024-09-26 16:00:30.621745] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337630621734, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621754] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292dcb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1841823684513746176, dag_cnt=25, dag_type_cnts=24) [2024-09-26 16:00:30.621756] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292dcb0) [2024-09-26 16:00:30.621758] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, create_flush_dag_time:1727337630621758, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405470, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:323}, checkpoint_unit_ptr:0x7f92a1a7d370}, is_freeze_clock:true}) [2024-09-26 16:00:30.621761] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337630621751, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621771] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f927292e1e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-2678407261265885087, dag_cnt=26, dag_type_cnts=25) [2024-09-26 16:00:30.621774] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292e1e0) [2024-09-26 16:00:30.621775] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, create_flush_dag_time:1727337630621775, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405487, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:377}, checkpoint_unit_ptr:0x7f92a1a7e310}, is_freeze_clock:true}) [2024-09-26 16:00:30.621779] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337630621769, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621788] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292e710, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7175089551615529855, dag_cnt=27, dag_type_cnts=26) [2024-09-26 16:00:30.621790] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292e710) [2024-09-26 16:00:30.621792] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, create_flush_dag_time:1727337630621792, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405503, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:297}, checkpoint_unit_ptr:0x7f92a1a7eae0}, is_freeze_clock:true}) [2024-09-26 16:00:30.621796] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337630621785, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621805] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292ec40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3730526293242960203, dag_cnt=28, dag_type_cnts=27) [2024-09-26 16:00:30.621807] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292ec40) [2024-09-26 16:00:30.621809] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337630621809, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405519, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101065}, checkpoint_unit_ptr:0x7f92a1a7db40}, is_freeze_clock:true}) [2024-09-26 16:00:30.621812] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337630621802, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621823] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927292f170, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=4663792628280492528, dag_cnt=29, dag_type_cnts=28) [2024-09-26 16:00:30.621825] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292f170) [2024-09-26 16:00:30.621826] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337630621826, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405536, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:333}, checkpoint_unit_ptr:0x7f92a3bc4850}, is_freeze_clock:true}) [2024-09-26 16:00:30.621830] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337630621819, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621855] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=21] add dag success(dag=0x7f927292f6a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8399557287176188414, dag_cnt=30, dag_type_cnts=29) [2024-09-26 16:00:30.621858] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f927292f6a0) [2024-09-26 16:00:30.621861] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337630621861, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405552, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101066}, checkpoint_unit_ptr:0x7f92a3bc5020}, is_freeze_clock:true}) [2024-09-26 16:00:30.621864] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337630621851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621874] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927292fbd0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8068938909778324349, dag_cnt=31, dag_type_cnts=30) [2024-09-26 16:00:30.621877] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292fbd0) [2024-09-26 16:00:30.621881] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, create_flush_dag_time:1727337630621881, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405569, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:334}, checkpoint_unit_ptr:0x7f92a3bc57f0}, is_freeze_clock:true}) [2024-09-26 16:00:30.621885] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337630621871, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621895] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272930100, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7665372330263152188, dag_cnt=32, dag_type_cnts=31) [2024-09-26 16:00:30.621898] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272930100) [2024-09-26 16:00:30.621899] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, create_flush_dag_time:1727337630621899, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405586, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:451}, checkpoint_unit_ptr:0x7f92a3bc4080}, is_freeze_clock:true}) [2024-09-26 16:00:30.621903] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337630621892, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621912] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272930630, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6759548225246904317, dag_cnt=33, dag_type_cnts=32) [2024-09-26 16:00:30.621914] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272930630) [2024-09-26 16:00:30.621916] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, create_flush_dag_time:1727337630621916, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405602, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:378}, checkpoint_unit_ptr:0x7f92a1a7f2b0}, is_freeze_clock:true}) [2024-09-26 16:00:30.621919] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337630621909, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621929] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272930b60, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8944614441376604790, dag_cnt=34, dag_type_cnts=33) [2024-09-26 16:00:30.621932] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272930b60) [2024-09-26 16:00:30.621933] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, create_flush_dag_time:1727337630621933, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209456003, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:121}, checkpoint_unit_ptr:0x7f92a3bc5fc0}, is_freeze_clock:true}) [2024-09-26 16:00:30.621937] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337630621926, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621946] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272931090, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=507863347806783454, dag_cnt=35, dag_type_cnts=34) [2024-09-26 16:00:30.621949] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272931090) [2024-09-26 16:00:30.621950] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, create_flush_dag_time:1727337630621950, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405633, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:215}, checkpoint_unit_ptr:0x7f92a3bc86d0}, is_freeze_clock:true}) [2024-09-26 16:00:30.621953] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337630621943, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621963] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f92729315c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1604282355669434429, dag_cnt=36, dag_type_cnts=35) [2024-09-26 16:00:30.621973] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f92729315c0) [2024-09-26 16:00:30.621977] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, create_flush_dag_time:1727337630621977, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405649, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:456}, checkpoint_unit_ptr:0x7f92a3bc6790}, is_freeze_clock:true}) [2024-09-26 16:00:30.621980] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337630621960, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.621991] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272931af0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3553649416163308074, dag_cnt=37, dag_type_cnts=36) [2024-09-26 16:00:30.621994] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272931af0) [2024-09-26 16:00:30.621995] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337630621995, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405666, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:154}, checkpoint_unit_ptr:0x7f92a3bc6f60}, is_freeze_clock:true}) [2024-09-26 16:00:30.621999] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337630621987, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.622008] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272932020, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8097406237500329300, dag_cnt=38, dag_type_cnts=37) [2024-09-26 16:00:30.622011] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272932020) [2024-09-26 16:00:30.622012] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337630622012, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405682, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101015}, checkpoint_unit_ptr:0x7f92a3bc7730}, is_freeze_clock:true}) [2024-09-26 16:00:30.622015] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337630622005, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.622025] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272932550, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3976350754940852513, dag_cnt=39, dag_type_cnts=38) [2024-09-26 16:00:30.622027] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272932550) [2024-09-26 16:00:30.622029] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337630622029, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405699, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101016}, checkpoint_unit_ptr:0x7f92a3bc7f00}, is_freeze_clock:true}) [2024-09-26 16:00:30.622032] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337630622022, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.622041] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272932a80, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8720304653345806796, dag_cnt=40, dag_type_cnts=39) [2024-09-26 16:00:30.622044] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272932a80) [2024-09-26 16:00:30.622045] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, create_flush_dag_time:1727337630622045, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405716, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101063}, checkpoint_unit_ptr:0x7f92a3bca610}, is_freeze_clock:true}) [2024-09-26 16:00:30.622049] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337630622039, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.622059] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272932fb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1830562711513853292, dag_cnt=41, dag_type_cnts=40) [2024-09-26 16:00:30.622061] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272932fb0) [2024-09-26 16:00:30.622063] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, create_flush_dag_time:1727337630622063, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405733, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:452}, checkpoint_unit_ptr:0x7f92a3bc9e40}, is_freeze_clock:true}) [2024-09-26 16:00:30.622066] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337630622055, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.622076] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f92729334e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1536604874905918360, dag_cnt=42, dag_type_cnts=41) [2024-09-26 16:00:30.622078] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f92729334e0) [2024-09-26 16:00:30.622080] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337630622080, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405750, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:328}, checkpoint_unit_ptr:0x7f92a3bc8ea0}, is_freeze_clock:true}) [2024-09-26 16:00:30.622083] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337630622073, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.622093] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272933a10, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6933628521298738662, dag_cnt=43, dag_type_cnts=42) [2024-09-26 16:00:30.622095] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272933a10) [2024-09-26 16:00:30.622097] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337630622097, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405767, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:330}, checkpoint_unit_ptr:0x7f92a3bc9670}, is_freeze_clock:true}) [2024-09-26 16:00:30.622100] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337630622090, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.622109] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272933f40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1056199536334423564, dag_cnt=44, dag_type_cnts=43) [2024-09-26 16:00:30.622112] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272933f40) [2024-09-26 16:00:30.622113] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337630622113, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405784, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:329}, checkpoint_unit_ptr:0x7f92a3bcb5b0}, is_freeze_clock:true}) [2024-09-26 16:00:30.622117] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337630622107, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.622126] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272934470, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7405463607918801432, dag_cnt=45, dag_type_cnts=44) [2024-09-26 16:00:30.622128] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272934470) [2024-09-26 16:00:30.622130] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, create_flush_dag_time:1727337630622130, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405801, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:332}, checkpoint_unit_ptr:0x7f92a3bcade0}, is_freeze_clock:true}) [2024-09-26 16:00:30.622133] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337630622123, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.622142] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f92729349a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6699723950958743903, dag_cnt=46, dag_type_cnts=45) [2024-09-26 16:00:30.622145] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f92729349a0) [2024-09-26 16:00:30.622149] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, create_flush_dag_time:1727337630622149, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405818, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101067}, checkpoint_unit_ptr:0x7f92a3bcbd80}, is_freeze_clock:true}) [2024-09-26 16:00:30.622153] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337630622140, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.622160] INFO [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:655) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] traversal_flush successfully(ls_->get_ls_id()={id:1}, flush_tasks=[{table:0x7f92a1a70850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a70080, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71020, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a717f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71fc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72790, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72f60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73730, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73f00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a746d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75670, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a74ea0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76610, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75e40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a775b0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76de0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a77d80, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7a490, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a79cc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a794f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78550, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78d20, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7ac60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7cba0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7b430, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7bc00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7c3d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7d370, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7e310, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7eae0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7db40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a3bc4850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}]) [2024-09-26 16:00:30.622209] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=45] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-09-26 16:00:30.631679] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:30.631800] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726163995122514000, tablet={this:0x7f92b862bfe0, is_inited:true, wash_score:1727337630620660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:505, tablet_addr:[816017-1849-0-2305843645944356864-2](offset=1761280,size=593,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:200489}, data_tablet_id:{id:200489}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725991206240736000, v:0}, clog_checkpoint_scn:{val:1726163995200406000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726163995122514000, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1725991205062061000, cur_report_version:1725991205062061000, data_checksum:1210484554, row_count:46137450}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1725120000848336, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726201924319685000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1725120000848336}, table_store_addr:{ptr:{this:0x7f92b862c674, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:4, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305877511761428480-0](offset=1703936,size=32133,type=2,seq=0), type:"MAJOR", tablet_id:{id:200489}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1725991205062061000, v:0}}, snapshot_version:1725991205062061000, max_merge_version:1725991203769806000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305849822107303301-0](offset=1736069,size=6345,type=2,seq=0), type:"MINOR", tablet_id:{id:200489}, scn_range:{start_scn:{val:1725991206240736000, v:0}, end_scn:{val:1726076389100953000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726076389025660000, upper_trans_version:1726076389025660000}, {i:1, addr:[816017-1849-0-2305844368572585550-0](offset=1742414,size=1266,type=2,seq=0), type:"MINI", tablet_id:{id:200489}, scn_range:{start_scn:{val:1726076389100953000, v:0}, end_scn:{val:1726077259218062001, v:0}}, contain_uncommitted_row:"yes", max_merge_version:1726077257041685000, upper_trans_version:1726077259766338000}, {i:2, addr:[816017-1849-0-2305843929412180800-0](offset=1743680,size=857,type=2,seq=0), type:"MINI", tablet_id:{id:200489}, scn_range:{start_scn:{val:1726077259218062001, v:0}, end_scn:{val:1726077600564631000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726077598368094000, upper_trans_version:1726077598368094000}, {i:3, addr:[816017-1849-0-2305849899416723097-0](offset=1744537,size=6417,type=2,seq=0), type:"MINOR", tablet_id:{id:200489}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726163995200406000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726163995200406000, upper_trans_version:1726163995200406000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337630620744, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843710368858112-2](offset=1753088,size=653,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843165981754119-0](offset=1753863,size=146,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255102325657-0](offset=1754009,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1725991205062061000}, medium_info_list:{ptr:(nil), addr:[816017-1849-0-2305843901494906880-0](offset=1757184,size=831,type=2,seq=0)}, auto_inc_seq:{ptr:0x7f92b862cafc, auto_inc_seq:{version:1, intervals_count:1, intervals:{tablet_id:{id:0}, start:55640001, end:9223372036854775807}}, addr:[816017-1849-0-2305843140211950221-0](offset=1753741,size=122,type=2,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1725120001569133000, v:0}, create_commit_version:1725120001569133000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a70080, 0x7f92a1a73730, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.631865] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:30.631878] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.633234] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1, storage_schema:{this:0x7f92728f12b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.633251] INFO [STORAGE] prepare_tx_data_list (ob_tx_data_memtable.cpp:266) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=15] construct list more than once. skip this time(this=0x7f92a2294080) [2024-09-26 16:00:30.636049] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.636081] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=31] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337630620777, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:30.636090] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.636201] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726158220122116000) [2024-09-26 16:00:30.636213] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=15] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:30.636217] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=8] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.636220] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.636238] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847223}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:30.636269] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=21] succeed to build merge ctx(tablet_id={id:1152921504606847223}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847223}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630620966, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=15.26ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.636583] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=28] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:30.637298] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=5] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=7999, total_rows=55993) [2024-09-26 16:00:30.637730] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=8] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92729274f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EC-0-0, dag_ret:0, dag_status:2, start_time:1727337630620938, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d470, output_block_cnt_arr:0x7f9264b1d478, concurrent_cnt:1, estimate_row_cnt:332829, estimate_occupy_size:150994944, latest_update_ts:1727337630620966, estimated_finish_time:1727337647637802}) [2024-09-26 16:00:30.637748] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=17] succeed to init merge ctx(task={this:0x7f92729278b0, type:15, status:2, dag:{ObIDag:{this:0x7f92729274f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EC-0-0, dag_ret:0, dag_status:2, start_time:1727337630620938, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.637756] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=7] task finish process(ret=0, start_time=1727337630620963, end_time=1727337630637756, runtime=16793, *this={this:0x7f92729278b0, type:15, status:2, dag:{ObIDag:{this:0x7f92729274f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EC-0-0, dag_ret:0, dag_status:2, start_time:1727337630620938, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.637796] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=1] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.637886] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:4, allocator:0x7f926366e7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:30.637914] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=23] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:4, allocator:0x7f926366f558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3e350, local_datums:0x7f9263a3e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:30.638156] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:30.638219] ERROR issue_dba_error (ob_log.cpp:1875) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.638224] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.638234] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=9][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:30.638237] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:30.638239] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9264d60140, hex: 3230323430393131303133333131, cstr: 20240911013311},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9264d6014e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d600e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d60118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-15038, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7227, data_checksum:1452123993, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9264dce030, buf_size:7227, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1818264584, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:00:30.638255] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=15][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:30.638257] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:30.638259] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:30.638265] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=5][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:30.638291] INFO [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:329) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=2] pre-process commit versions row(get_ls_id()={id:1}, do_recycle_=true, current_time=1727337630638289, prev_recycle_time=0) [2024-09-26 16:00:30.638267] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f926366e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926366e118, limit_param:null, stmt_allocator:0x7f926366e7d8, allocator:0x7f926366e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:00:30.638348] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=80][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:30.638350] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f926366e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926366e118, limit_param:null, stmt_allocator:0x7f926366e7d8, allocator:0x7f926366e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:00:30.638384] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=33][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:30.638387] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=1] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.638402] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=10] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337630620777, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.638412] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:30.638427] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=3][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:30.638431] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:30.638437] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=5] task finish process(ret=-4184, start_time=1727337630637815, end_time=1727337630638436, runtime=621, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f92729274f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EC-0-0, dag_ret:0, dag_status:2, start_time:1727337630620938, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.638449] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=12][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f92729274f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EC-0-0, dag_ret:0, dag_status:2, start_time:1727337630620938, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:30.638462] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=8] dag finished(dag_ret=-4184, runtime=17524, dag_cnt=45, dag_cnts_[dag.get_type()]=44, &dag=0x7f92729274f0, dag={ObIDag:{this:0x7f92729274f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EC-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630620938, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.638473] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EC-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337630620936, task_id:YB42C0A8002D-0006230056D8C5EC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.638516] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337630638512, task_id:YB42C0A8002D-0006230056D8C5EE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.638523] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=6] schedule one task(task=0x7f9272928310, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.638691] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.638699] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.638701] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.638708] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.638761] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.638775] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=13] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337630620761, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:30.638783] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.638884] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:30.638889] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.638898] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847222}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:30.638921] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:1152921504606847222}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847222}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630621063, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=17.83ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.639175] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=37] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:30.639928] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=8366, total_rows=58562) [2024-09-26 16:00:30.640350] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=8] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272926fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EB-0-0, dag_ret:0, dag_status:2, start_time:1727337630620931, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad470, output_block_cnt_arr:0x7f92684ad478, concurrent_cnt:1, estimate_row_cnt:329211, estimate_occupy_size:150994944, latest_update_ts:1727337630621063, estimated_finish_time:1727337647640421}) [2024-09-26 16:00:30.640365] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=14] succeed to init merge ctx(task={this:0x7f9272927380, type:15, status:2, dag:{ObIDag:{this:0x7f9272926fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EB-0-0, dag_ret:0, dag_status:2, start_time:1727337630620931, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.640373] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=7] task finish process(ret=0, start_time=1727337630621062, end_time=1727337630640373, runtime=19311, *this={this:0x7f9272927380, type:15, status:2, dag:{ObIDag:{this:0x7f9272926fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EB-0-0, dag_ret:0, dag_status:2, start_time:1727337630620931, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.640410] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337630620836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.640424] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752755184, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828752755184, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.640439] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.640442] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.640447] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:226}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752755184, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.640479] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:226}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752755184, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:226}, table_count:1, [{i:0, table_key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}, merge_scn:{val:1726241777443612000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630621085, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=19.36ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.640504] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272927a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5ED-0-0, dag_ret:0, dag_status:2, start_time:1727337630620946, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}, scanned_row_cnt_arr:0x7f9264b23480, output_block_cnt_arr:0x7f9264b23488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:56623104, latest_update_ts:1727337630621085, estimated_finish_time:1727337647640531}) [2024-09-26 16:00:30.640509] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272927de0, type:15, status:2, dag:{ObIDag:{this:0x7f9272927a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5ED-0-0, dag_ret:0, dag_status:2, start_time:1727337630620946, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:30.640514] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=5] task finish process(ret=0, start_time=1727337630621083, end_time=1727337630640514, runtime=19431, *this={this:0x7f9272927de0, type:15, status:2, dag:{ObIDag:{this:0x7f9272927a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5ED-0-0, dag_ret:0, dag_status:2, start_time:1727337630620946, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:30.640587] INFO [STORAGE] get_ls_min_end_scn (ob_ls_tablet_service.cpp:5618) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=9] get ls min end scn finish(ls_checkpoint={val:1726163586617842000, v:0}) [2024-09-26 16:00:30.640597] WDIAG [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:761) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=5][errcode=-4023] failed to get snapshot version(ret=-4023, MTL_ID()=1) [2024-09-26 16:00:30.640598] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=3] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.640601] INFO [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:773) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=4] get tx data recycle scn finish.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, recycle_scn={val:1726076389100953000, v:0}, min_end_scn_from_old_tablets={val:4611686018427387903, v:0}, min_end_scn_from_latest_tablets={val:1726076389100953000, v:0}) [2024-09-26 16:00:30.640605] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=5] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.640607] WDIAG [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:336) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=5][errcode=-4023] get recycle ts failed.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:30.640612] WDIAG [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:236) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=2][errcode=-4023] process commit version row failed.(ret=-4023, ret="OB_EAGAIN", this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337630620632, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:30.640623] INFO [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:254) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=11] [TX DATA MERGE]pre process for merge done(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}, this=0x7f92a2294080, tg=time guard have no click events for optimization, fake_tx_data_guard={tx_data:{tx_id:{txid:9223372036854775807}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337630620632, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:30.640639] WDIAG [STORAGE.COMPACTION] pre_process_tx_data_table_merge_ (ob_tx_table_merge_task.cpp:117) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=15][errcode=-4023] do pre process for tx data table merge failed.(ret=-4023, ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, table={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337630620632, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:30.640648] WDIAG [STORAGE.COMPACTION] inner_init_ctx (ob_tx_table_merge_task.cpp:157) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=9][errcode=-4023] pre process tx data table for merge failed.(ret=-4023, ret="OB_EAGAIN", ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.640652] WDIAG [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1061) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=3][errcode=-4023] fail to inner init ctx(ret=-4023, tablet_id={id:49402}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, create_snapshot_version:0, is_full_merge:false, merge_level:1, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:4, range_array:[], concurrent_cnt:0, is_inited:false}, schema_ctx:{schema_version:1, storage_schema:{this:0x7f92728f12b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:49402}, table_count:1, [{i:0, table_key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref:4}]}, schedule_major:false, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}, merge_scn:{val:4611686018427387903, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:NULL, compaction_filter:NULL, time_guard:total=19.70ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}) [2024-09-26 16:00:30.640673] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:975) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=21][errcode=-4023] failed to build merge ctx(ret=-4023, ctx->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.640676] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:994) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=3][errcode=-4023] sstable merge finish(ret=-4023, ctx=0x7f92728f0060, task={this:0x7f92729263f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272926030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C5E8-0-0, dag_ret:0, dag_status:2, start_time:1727337630620906, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:30.640684] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=8][errcode=-4023] failed to process task(ret=-4023) [2024-09-26 16:00:30.640686] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=2] task finish process(ret=-4023, start_time=1727337630620949, end_time=1727337630640686, runtime=19737, *this={this:0x7f92729263f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272926030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C5E8-0-0, dag_ret:0, dag_status:2, start_time:1727337630620906, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:30.640677] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d36060, count:4, allocator:0x7f92c244a7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d36350, local_datums:0x7f9287d36350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:30.640677] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=226, range=0x7f92a9952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:5, allocator:0x7f9269f527d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665f8360, local_datums:0x7f92665f8360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:30.640692] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=6][errcode=-4023] failed to do work(ret=-4023, *task_={this:0x7f92729263f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272926030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C5E8-0-0, dag_ret:0, dag_status:2, start_time:1727337630620906, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:30.640698] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=18] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a66060, count:4, allocator:0x7f92c244b558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a66350, local_datums:0x7f9264a66350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:30.640701] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=6] dag finished(dag_ret=-4023, runtime=19795, dag_cnt=44, dag_cnts_[dag.get_type()]=0, &dag=0x7f9272926030, dag={ObIDag:{this:0x7f9272926030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C5E8-0-0, dag_ret:-4023, dag_status:5, start_time:1727337630620906, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}) [2024-09-26 16:00:30.640711] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C5E8-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337630620897, task_id:YB42C0A8002D-0006230056D8C5E8-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.640830] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.640844] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=14] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337630620744, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=17, max_schema_version_on_memtable=1725120002010696) [2024-09-26 16:00:30.640852] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=17, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.640955] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=7] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1887, total_rows=92463) [2024-09-26 16:00:30.640956] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=7][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:30.640959] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.640966] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200489}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:19, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.640986] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:200489}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200489}, table_count:2, [{i:0, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630621006, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=19.96ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.641016] ERROR issue_dba_error (ob_log.cpp:1875) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.641020] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.641029] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=9][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:30.641032] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:30.641038] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=6][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9287dd6140, hex: 3230323430393131303133333438, cstr: 20240911013348},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9287dd614e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd60e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd6118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3080, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7221, data_checksum:4284205003, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265b00030, buf_size:7221, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1004563202, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:00:30.641054] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=16][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:30.641057] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=3][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:30.641059] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:30.641061] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:30.641063] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d36350, local_datums:0x7f9287d36350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d363d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d36408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d36350, local_datums:0x7f9287d36350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d363d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d36408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92c244a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c244a118, limit_param:null, stmt_allocator:0x7f92c244a7d8, allocator:0x7f92c244a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4048785788651384883, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d36350, local_datums:0x7f9287d36350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d363d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d36408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:00:30.641126] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=63][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:30.641129] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=3][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d36350, local_datums:0x7f9287d36350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d363d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d36408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92c244a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c244a118, limit_param:null, stmt_allocator:0x7f92c244a7d8, allocator:0x7f92c244a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4048785788651384883, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d36350, local_datums:0x7f9287d36350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d363d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d36408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:00:30.641162] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=32][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:30.641165] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=2] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.641178] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=9] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337630620761, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.641188] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:30.641200] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:30.641204] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:30.641206] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337630640617, end_time=1727337630641205, runtime=588, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9272926fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EB-0-0, dag_ret:0, dag_status:2, start_time:1727337630620931, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.641213] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9272926fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EB-0-0, dag_ret:0, dag_status:2, start_time:1727337630620931, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:30.641221] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=20289, dag_cnt=43, dag_cnts_[dag.get_type()]=43, &dag=0x7f9272926fc0, dag={ObIDag:{this:0x7f9272926fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EB-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630620931, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.641229] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EB-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337630620929, task_id:YB42C0A8002D-0006230056D8C5EB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.641259] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337630641256, task_id:YB42C0A8002D-0006230056D8C5EF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.641265] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=5] schedule one task(task=0x7f9272928840, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.641274] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337630641273, task_id:YB42C0A8002D-0006230056D8C5F0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.641276] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=2] schedule one task(task=0x7f9272928d70, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.641283] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272926a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EA-0-0, dag_ret:0, dag_status:2, start_time:1727337630620926, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515460, output_block_cnt_arr:0x7f9268515468, concurrent_cnt:1, estimate_row_cnt:254187, estimate_occupy_size:138412032, latest_update_ts:1727337630621006, estimated_finish_time:1727337647641348}) [2024-09-26 16:00:30.641293] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=10] succeed to init merge ctx(task={this:0x7f9272926e50, type:15, status:2, dag:{ObIDag:{this:0x7f9272926a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EA-0-0, dag_ret:0, dag_status:2, start_time:1727337630620926, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.641299] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=5] task finish process(ret=0, start_time=1727337630621005, end_time=1727337630641299, runtime=20294, *this={this:0x7f9272926e50, type:15, status:2, dag:{ObIDag:{this:0x7f9272926a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EA-0-0, dag_ret:0, dag_status:2, start_time:1727337630620926, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.641309] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=2] schedule one task(task=0x7f9262c043c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.641373] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5c060, count:3, allocator:0x7f92691a67d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263b5c320, local_datums:0x7f9263b5c320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:30.641390] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=13] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:3, allocator:0x7f92691a7558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:30.641566] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=7][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:30.641614] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.641620] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.641626] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:30.641628] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:30.641630] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62070, hex: 9F094D0300000000, int: 55380383},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d620a8, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d620e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3044, column_count:19, rowkey_column_count:3, has_column_checksum:0, row_count:83, row_store_type:0, opt:5, var_column_count:0, row_offset:16109, original_length:16381, max_merged_trans_version:1726163995623997000, data_length:16381, data_zlength:7759, data_checksum:3416849541, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92729ee030, buf_size:7759, data_size:16381, row_count:83, column_count:19, max_merged_trans_version:1726163995623997000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3537734793, row_count_delta:83, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16381}) [2024-09-26 16:00:30.641642] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=12][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:30.641644] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:30.641630] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.641646] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:30.641647] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:30.641647] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=16] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337630620709, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=18, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:30.641655] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.641649] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f92691a6e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691a6e98, limit_param:null, stmt_allocator:0x7f92691a7558, allocator:0x7f92691a7558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337630620744, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}) [2024-09-26 16:00:30.641713] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=64][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:30.641714] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f92691a6e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691a6e98, limit_param:null, stmt_allocator:0x7f92691a7558, allocator:0x7f92691a7558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337630620744, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}]) [2024-09-26 16:00:30.641752] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=38][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:30.641754] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.641764] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=8] partition merge iter row count(i=1, row_count=83, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, table={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337630620744, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.641785] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=18][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:30.641795] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:30.641798] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:30.641800] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337630641319, end_time=1727337630641800, runtime=481, *this={this:0x7f9262c043c0, type:1, status:2, dag:{ObIDag:{this:0x7f9272926a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EA-0-0, dag_ret:0, dag_status:2, start_time:1727337630620926, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.641807] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262c043c0, type:1, status:2, dag:{ObIDag:{this:0x7f9272926a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EA-0-0, dag_ret:0, dag_status:2, start_time:1727337630620926, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:30.641814] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=20888, dag_cnt=42, dag_cnts_[dag.get_type()]=42, &dag=0x7f9272926a90, dag={ObIDag:{this:0x7f9272926a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EA-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630620926, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.641822] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EA-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630620923, task_id:YB42C0A8002D-0006230056D8C5EA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.641858] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630641856, task_id:YB42C0A8002D-0006230056D8C5F1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.641861] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=3] schedule one task(task=0x7f92729292a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.641872] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:30.641878] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.641884] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.641888] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.641891] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.641896] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.641885] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200240}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.641906] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:200240}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200240}, table_count:2, [{i:0, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8637b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630621039, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=20.84ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.641901] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861eec0, is_inited:true, wash_score:1727337630620660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:493, tablet_addr:[816017-1849-0-2305843633059586048-2](offset=1892352,size=581,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1}, data_tablet_id:{id:1}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:778798285, row_count:7249}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861f59c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844357835309056-0](offset=1884160,size=1256,type=2,seq=0), type:"MAJOR", tablet_id:{id:1}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077591663360001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843964845802728-0](offset=1885416,size=890,type=2,seq=0), type:"MINOR", tablet_id:{id:1}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178578410947000, upper_trans_version:1726178578410947000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337630621445, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcd4f0, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241621541056000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405180}, this:0x7f92a3bcd4f0, timestamp:1727334209405180, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:53, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277807268534000, v:0}, rec_scn:{val:1726241631569302000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771609600-2](offset=1888256,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843169203114317-0](offset=1888589,size=149,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881234914-0](offset=1888738,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828751995331000, v:0}, create_commit_version:1710828751995331000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a76610, 0x7f92a3bcd4f0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.641947] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=46] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.641958] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241621541056000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.642151] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=19] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:30.642569] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272926560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E9-0-0, dag_ret:0, dag_status:2, start_time:1727337630620919, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f460, output_block_cnt_arr:0x7f9264b1f468, concurrent_cnt:1, estimate_row_cnt:85417, estimate_occupy_size:134217728, latest_update_ts:1727337630621039, estimated_finish_time:1727337647642632}) [2024-09-26 16:00:30.642581] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=11] succeed to init merge ctx(task={this:0x7f9272926920, type:15, status:2, dag:{ObIDag:{this:0x7f9272926560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E9-0-0, dag_ret:0, dag_status:2, start_time:1727337630620919, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.642599] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=18] task finish process(ret=0, start_time=1727337630621038, end_time=1727337630642599, runtime=21561, *this={this:0x7f9272926920, type:15, status:2, dag:{ObIDag:{this:0x7f9272926560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E9-0-0, dag_ret:0, dag_status:2, start_time:1727337630620919, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.642623] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=4] schedule one task(task=0x7f9287d5e3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.642627] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:30.642631] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.642715] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92aa6d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665aa060, count:3, allocator:0x7f92c4a5e7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665aa320, local_datums:0x7f92665aa320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:30.642740] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=17] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92aa6d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684e2060, count:3, allocator:0x7f92c4a5f558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92684e2320, local_datums:0x7f92684e2320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:30.642942] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:30.642986] ERROR issue_dba_error (ob_log.cpp:1875) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.642989] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf2041c8 0xf217c49 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.642995] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:30.642998] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:30.643000] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9268596108, hex: 313030303030333830, cstr: 100000380},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685960a8, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685960e0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-29413, column_count:20, rowkey_column_count:3, has_column_checksum:0, row_count:100, row_store_type:0, opt:0, var_column_count:0, row_offset:16082, original_length:16422, max_merged_trans_version:1726189672100745000, data_length:16454, data_zlength:5272, data_checksum:4171874375, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9268428030, buf_size:5272, data_size:16454, row_count:100, column_count:20, max_merged_trans_version:1726189672100745000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:371002229, row_count_delta:22, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16422}) [2024-09-26 16:00:30.643012] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=12][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:30.643014] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:30.643016] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:30.643018] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:30.643020] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1526) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, partition_fuser_->get_result_row()={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665a8148, local_datums:0x7f92665a8148}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665a8190, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665a81c8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665a83c0, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665a83f8, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, fuse_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665aa320, local_datums:0x7f92665aa320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92c4a5e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c4a5e118, limit_param:null, stmt_allocator:0x7f92c4a5e7d8, allocator:0x7f92c4a5e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665aa320, local_datums:0x7f92665aa320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92684e2320, local_datums:0x7f92684e2320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e23a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e25d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92c4a5ee98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c4a5ee98, limit_param:null, stmt_allocator:0x7f92c4a5f558, allocator:0x7f92c4a5f558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92684e2320, local_datums:0x7f92684e2320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e23a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e25d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337630620709, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:30.643124] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=103][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665aa320, local_datums:0x7f92665aa320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92c4a5e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c4a5e118, limit_param:null, stmt_allocator:0x7f92c4a5e7d8, allocator:0x7f92c4a5e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665aa320, local_datums:0x7f92665aa320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92684e2320, local_datums:0x7f92684e2320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e23a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e25d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92c4a5ee98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c4a5ee98, limit_param:null, stmt_allocator:0x7f92c4a5f558, allocator:0x7f92c4a5f558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92684e2320, local_datums:0x7f92684e2320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e23a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e25d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337630620709, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:30.643199] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=75][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:30.643202] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=2] partition merge iter row count(i=0, row_count=23, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.643213] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=8] partition merge iter row count(i=1, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337630620709, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.643222] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:30.643246] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:30.643251] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=5][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:30.643253] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337630642669, end_time=1727337630643253, runtime=584, *this={this:0x7f9287d5e3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9272926560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E9-0-0, dag_ret:0, dag_status:2, start_time:1727337630620919, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.643261] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287d5e3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9272926560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E9-0-0, dag_ret:0, dag_status:2, start_time:1727337630620919, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:30.643269] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=22349, dag_cnt=41, dag_cnts_[dag.get_type()]=41, &dag=0x7f9272926560, dag={ObIDag:{this:0x7f9272926560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5E9-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630620919, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.643277] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5E9-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630620914, task_id:YB42C0A8002D-0006230056D8C5E9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.643308] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337630643306, task_id:YB42C0A8002D-0006230056D8C5F2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.643314] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=6] schedule one task(task=0x7f92729297d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.643344] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.643348] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.643350] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.643353] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.643357] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.643364] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.644391] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.644397] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.644399] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.644413] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=11] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.644433] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.644436] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.644437] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.644440] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.644442] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.644447] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.647131] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=19][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:30.647180] ERROR issue_dba_error (ob_log.cpp:1875) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=10][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.647183] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.647190] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:30.647192] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:30.647194] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684ac070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684ac0a8, hex: 0000000000000000, int: 0},idx=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92684ac0e0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684ac118, hex: 58A3C56A4A280BE8, int: -1726241732082425000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684ac150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-19590, column_count:10, rowkey_column_count:5, has_column_checksum:0, row_count:224, row_store_type:0, opt:0, var_column_count:0, row_offset:15527, original_length:16363, max_merged_trans_version:1726241777443612000, data_length:16363, data_zlength:16363, data_checksum:2057046089, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926f204090, buf_size:16363, data_size:16363, row_count:224, column_count:10, max_merged_trans_version:1726241777443612000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1006468742, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16363}) [2024-09-26 16:00:30.647207] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=13][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:30.647209] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:30.647211] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:30.647212] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:30.647214] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:30.647216] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:30.647220] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=4][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665f8360, local_datums:0x7f92665f8360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f83a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665f83e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665f8360, local_datums:0x7f92665f8360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f83a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665f83e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f9269f52118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269f52118, limit_param:null, stmt_allocator:0x7f9269f527d8, allocator:0x7f9269f527d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665f8360, local_datums:0x7f92665f8360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f83a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665f83e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337630620836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}) [2024-09-26 16:00:30.647300] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=80][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:30.647302] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665f8360, local_datums:0x7f92665f8360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f83a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665f83e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f9269f52118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269f52118, limit_param:null, stmt_allocator:0x7f9269f527d8, allocator:0x7f9269f527d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665f8360, local_datums:0x7f92665f8360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f83a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665f83e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337630620836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}]) [2024-09-26 16:00:30.647342] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=40][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:30.647345] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2] partition merge iter row count(i=0, row_count=224, ghost_row_count=0, pkey={tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, table={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337630620836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.647367] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=18][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:30.647377] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:30.647380] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:30.647382] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337630640618, end_time=1727337630647382, runtime=6764, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9272927a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5ED-0-0, dag_ret:0, dag_status:2, start_time:1727337630620946, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:30.647367] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=29] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b863bfe0, is_inited:true, wash_score:1727337630620660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:506, tablet_addr:[816017-1849-0-2305843647017062400-1](offset=724992,size=594,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, data_tablet_id:{id:200489}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725991206240736000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1725991205062061000, cur_report_version:1725991205062061000, data_checksum:2143506547, row_count:46137450}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1725120000848352, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1725120000848352}, table_store_addr:{ptr:null, addr:[816017-1849-0-2305845196426506240-1](offset=716800,size=2037,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718137333-0](offset=718837,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255101290641-0](offset=718993,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1725991205062061000}, medium_info_list:{ptr:(nil), addr:[816017-1849-0-2305843520315523072-0](offset=720896,size=476,type=2,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1725120001569133000, v:0}, create_commit_version:1725120001569133000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a746d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:30.647389] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9272927a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5ED-0-0, dag_ret:0, dag_status:2, start_time:1727337630620946, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:30.647394] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=27] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:30.647397] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=26450, dag_cnt=40, dag_cnts_[dag.get_type()]=40, &dag=0x7f9272927a20, dag={ObIDag:{this:0x7f9272927a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5ED-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630620946, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}) [2024-09-26 16:00:30.647403] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.647404] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5ED-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630620943, task_id:YB42C0A8002D-0006230056D8C5ED-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.647441] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337630647438, task_id:YB42C0A8002D-0006230056D8C5F3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.647445] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=4] schedule one task(task=0x7f9272929d00, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.647492] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.647497] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.647500] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.647504] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.647509] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.647517] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.650248] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:30.650257] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337630650246) [2024-09-26 16:00:30.650262] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337630549659, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:30.650275] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223359885734, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.650280] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:30.650287] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:30.650292] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:30.650295] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:30.650301] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.650303] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.650305] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337630650264) [2024-09-26 16:00:30.650314] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:30.651321] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337630621445, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.651344] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=23] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:0, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.651359] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.651372] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=12] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.651379] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:0, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.651399] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:1}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:0, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1}, table_count:1, [{i:0, table_key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}, merge_scn:{val:1726241621541056000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630641877, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=9.50ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.651425] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272928ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F1-0-0, dag_ret:0, dag_status:2, start_time:1727337630641861, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}, scanned_row_cnt_arr:0x7f92684b1480, output_block_cnt_arr:0x7f92684b1488, concurrent_cnt:1, estimate_row_cnt:3, estimate_occupy_size:6291456, latest_update_ts:1727337630641877, estimated_finish_time:1727337647651427}) [2024-09-26 16:00:30.651442] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=17] succeed to init merge ctx(task={this:0x7f92729292a0, type:15, status:2, dag:{ObIDag:{this:0x7f9272928ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F1-0-0, dag_ret:0, dag_status:2, start_time:1727337630641861, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:30.651448] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=5] task finish process(ret=0, start_time=1727337630641874, end_time=1727337630651447, runtime=9573, *this={this:0x7f92729292a0, type:15, status:2, dag:{ObIDag:{this:0x7f9272928ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F1-0-0, dag_ret:0, dag_status:2, start_time:1727337630641861, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:30.651513] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337630621487, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.651523] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753247280, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828753247280, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.651532] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.651536] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.651545] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:30.651540] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:412}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753247280, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.651556] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=7] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.651570] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=3] schedule one task(task=0x7f9268486030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.651556] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:412}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753247280, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:412}, table_count:1, [{i:0, table_key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630643338, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=8.20ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.651585] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=24] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272929410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F2-0-0, dag_ret:0, dag_status:2, start_time:1727337630643314, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1470, output_block_cnt_arr:0x7f92728f1478, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337630643338, estimated_finish_time:1727337647651587}) [2024-09-26 16:00:30.651591] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f92729297d0, type:15, status:2, dag:{ObIDag:{this:0x7f9272929410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F2-0-0, dag_ret:0, dag_status:2, start_time:1727337630643314, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.651596] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=5] task finish process(ret=0, start_time=1727337630643336, end_time=1727337630651596, runtime=8260, *this={this:0x7f92729297d0, type:15, status:2, dag:{ObIDag:{this:0x7f9272929410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F2-0-0, dag_ret:0, dag_status:2, start_time:1727337630643314, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.651637] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1, range=0x7f92a9d52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:8}, cols_desc:{ObIArray:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b56060, count:5, allocator:0x7f92b323e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:30.651764] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337630620897, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.651776] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828761605424, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828761605424, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.651785] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.651788] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.651794] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200004}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1710828761605424, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.651819] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=20] succeed to build merge ctx(tablet_id={id:200004}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828761605424, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200004}, table_count:1, [{i:0, table_key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8621da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630644430, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=7.36ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.651838] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92729289b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F0-0-0, dag_ret:0, dag_status:2, start_time:1727337630641276, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f460, output_block_cnt_arr:0x7f9264b1f468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337630644430, estimated_finish_time:1727337647651840}) [2024-09-26 16:00:30.651843] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272928d70, type:15, status:2, dag:{ObIDag:{this:0x7f92729289b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F0-0-0, dag_ret:0, dag_status:2, start_time:1727337630641276, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.651848] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=5] task finish process(ret=0, start_time=1727337630644429, end_time=1727337630651848, runtime=7419, *this={this:0x7f9272928d70, type:15, status:2, dag:{ObIDag:{this:0x7f92729289b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F0-0-0, dag_ret:0, dag_status:2, start_time:1727337630641276, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.652034] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=29][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:30.652078] ERROR issue_dba_error (ob_log.cpp:1875) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.652082] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.652089] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:30.652093] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:30.652095] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 17, flag: 0, null: 0, ptr: 0x7f9263b5e178, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e0a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 15, flag: 0, null: 0, ptr: 0x7f9263b5e189, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e118, hex: 60D2640CCC290BE8, int: -1726240075808452000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-20032, column_count:8, rowkey_column_count:5, has_column_checksum:0, row_count:158, row_store_type:0, opt:0, var_column_count:0, row_offset:15849, original_length:16421, max_merged_trans_version:1726241621541056000, data_length:16421, data_zlength:16421, data_checksum:3599888316, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926fc04090, buf_size:16421, data_size:16421, row_count:158, column_count:8, max_merged_trans_version:1726241621541056000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1714289658, row_count_delta:2, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16421}) [2024-09-26 16:00:30.652117] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=21][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:30.652120] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:30.652122] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:30.652123] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:30.652128] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=4][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:30.652130] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:30.652131] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f92b323e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b323e118, limit_param:null, stmt_allocator:0x7f92b323e7d8, allocator:0x7f92b323e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337630621445, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:30.652187] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=56][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:30.652189] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f92b323e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b323e118, limit_param:null, stmt_allocator:0x7f92b323e7d8, allocator:0x7f92b323e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337630621445, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:30.652224] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=34][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:30.652226] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=1] partition merge iter row count(i=0, row_count=158, ghost_row_count=0, pkey={tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, table={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337630621445, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.652238] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:30.652248] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:30.652251] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:30.652253] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337630651583, end_time=1727337630652253, runtime=670, *this={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9272928ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F1-0-0, dag_ret:0, dag_status:2, start_time:1727337630641861, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:30.652260] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9272928ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F1-0-0, dag_ret:0, dag_status:2, start_time:1727337630641861, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:30.652268] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=6] dag finished(dag_ret=-4184, runtime=10407, dag_cnt=39, dag_cnts_[dag.get_type()]=39, &dag=0x7f9272928ee0, dag={ObIDag:{this:0x7f9272928ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F1-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630641861, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}) [2024-09-26 16:00:30.652276] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F1-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337630641856, task_id:YB42C0A8002D-0006230056D8C5F1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.652304] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=4] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.652310] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=3] schedule one task(task=0x7f92684863c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.652318] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337630652317, task_id:YB42C0A8002D-0006230056D8C5F4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.652322] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=3] schedule one task(task=0x7f927292a230, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.652373] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=412, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:7}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a36060, count:4, allocator:0x7f92636087d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9263a36350, local_datums:0x7f9263a36350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:30.652374] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=200004, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b2060, count:3, allocator:0x7f926360a7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b2320, local_datums:0x7f92684b2320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:30.652392] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=11][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:30.652425] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.652427] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.652434] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:30.652436] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:30.652438] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 23, flag: 0, null: 0, ptr: 0x7f9263a3e140, hex: 6865617274626561745F736572766963655F65706F6368, cstr: heartbeat_service_epoch},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e0e0, hex: 8808AF9C9C5C0BE8, int: -1726184204453083000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-7627, column_count:7, rowkey_column_count:4, has_column_checksum:0, row_count:1, row_store_type:0, opt:5, var_column_count:0, row_offset:136, original_length:80, max_merged_trans_version:1726184204453083000, data_length:80, data_zlength:80, data_checksum:1549639079, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92b6804090, buf_size:80, data_size:80, row_count:1, column_count:7, max_merged_trans_version:1726184204453083000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:37835160, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:80}) [2024-09-26 16:00:30.652447] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=9][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:30.652449] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=2][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:00:30.652451] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=1][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:00:30.652452] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=1][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:00:30.652456] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=4][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:00:30.652458] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337630621487, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.652469] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:30.652475] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:30.652477] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=2][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:30.652479] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337630652331, end_time=1727337630652479, runtime=148, *this={this:0x7f92684863c0, type:1, status:2, dag:{ObIDag:{this:0x7f9272929410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F2-0-0, dag_ret:0, dag_status:2, start_time:1727337630643314, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.652484] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=5][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f92684863c0, type:1, status:2, dag:{ObIDag:{this:0x7f9272929410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F2-0-0, dag_ret:0, dag_status:2, start_time:1727337630643314, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:30.652510] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=23] dag finished(dag_ret=-4184, runtime=9196, dag_cnt=38, dag_cnts_[dag.get_type()]=38, &dag=0x7f9272929410, dag={ObIDag:{this:0x7f9272929410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F2-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630643314, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.652517] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F2-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630643306, task_id:YB42C0A8002D-0006230056D8C5F2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.652535] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=13][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:30.652552] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.652555] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.652557] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.652562] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.652565] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.652571] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241316899724000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.652574] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.652576] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.652583] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:30.652584] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337630652582, task_id:YB42C0A8002D-0006230056D8C5F5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.652585] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:30.652587] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=2] schedule one task(task=0x7f927292a760, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.652587] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e0a8, hex: C852A9F488570BE8, int: -1726189786434547000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e0e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19071, column_count:8, rowkey_column_count:3, has_column_checksum:0, row_count:218, row_store_type:0, opt:0, var_column_count:0, row_offset:15549, original_length:16361, max_merged_trans_version:1726190661653507000, data_length:16361, data_zlength:5996, data_checksum:2604580500, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92684de030, buf_size:5996, data_size:16361, row_count:218, column_count:8, max_merged_trans_version:1726190661653507000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:83488362, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16361}) [2024-09-26 16:00:30.652606] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=19][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:30.652608] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:30.652610] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:30.652611] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:30.652612] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b2320, local_datums:0x7f92684b2320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b23a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b2320, local_datums:0x7f92684b2320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b23a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f926360a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926360a118, limit_param:null, stmt_allocator:0x7f926360a7d8, allocator:0x7f926360a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b2320, local_datums:0x7f92684b2320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b23a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337630620897, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:30.652663] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=51][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:30.652664] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b2320, local_datums:0x7f92684b2320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b23a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f926360a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926360a118, limit_param:null, stmt_allocator:0x7f926360a7d8, allocator:0x7f926360a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b2320, local_datums:0x7f92684b2320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b23a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337630620897, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:30.652699] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=34][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:30.652701] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=1] partition merge iter row count(i=0, row_count=218, ghost_row_count=0, pkey={tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337630620897, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.652712] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:30.652722] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:30.652725] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:30.652727] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337630652327, end_time=1727337630652727, runtime=400, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f92729289b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F0-0-0, dag_ret:0, dag_status:2, start_time:1727337630641276, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.652733] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f92729289b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F0-0-0, dag_ret:0, dag_status:2, start_time:1727337630641276, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:30.652740] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=11464, dag_cnt=37, dag_cnts_[dag.get_type()]=37, &dag=0x7f92729289b0, dag={ObIDag:{this:0x7f92729289b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F0-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630641276, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.652747] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F0-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630641273, task_id:YB42C0A8002D-0006230056D8C5F0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.652774] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630652772, task_id:YB42C0A8002D-0006230056D8C5F6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.652777] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=3] schedule one task(task=0x7f927292ac90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.652784] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.652787] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.652789] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.652793] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.652803] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.652807] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.652809] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.652812] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.652815] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.652822] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.652797] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8633ce0, is_inited:true, wash_score:1727337630620660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[816017-1849-0-2305843644870795264-2](offset=1941504,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:331}, data_tablet_id:{id:331}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1433772493, row_count:436}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86343bc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844404006256640-0](offset=1933312,size=1299,type=2,seq=0), type:"MAJOR", tablet_id:{id:331}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726076731692125000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843945518499091-0](offset=1934611,size=872,type=2,seq=0), type:"MINOR", tablet_id:{id:331}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178440167442000, upper_trans_version:1726178440167442000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337630621542, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd32b0, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241465393532000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209503876}, this:0x7f92a3bd32b0, timestamp:1727334209503876, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:64, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277479152983000, v:0}, rec_scn:{val:1726242365923499000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843367845400576-2](offset=1937408,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719356238-0](offset=1937742,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881284074-0](offset=1937898,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447804108-0](offset=1938124,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a77d80, 0x7f92a3bd32b0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.652839] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=41] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.652845] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241465393532000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.652957] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337630620851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.652968] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=10] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:30.652971] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92728f3480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.652986] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337630621505, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.652997] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753327496, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753327496, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.653011] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.653015] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.653028] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337630296333]}) [2024-09-26 16:00:30.653020] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:444}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753327496, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.653080] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337630621542, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.653107] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=27] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.653120] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.653124] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.653125] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3214, total_rows=22498) [2024-09-26 16:00:30.653120] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=73] succeed to build merge ctx(tablet_id={id:444}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753327496, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:444}, table_count:1, [{i:0, table_key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8632d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630647485, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.53ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.653129] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.653129] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:331}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.653146] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272929940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F3-0-0, dag_ret:0, dag_status:2, start_time:1727337630647445, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad490, output_block_cnt_arr:0x7f92684ad498, concurrent_cnt:1, estimate_row_cnt:4, estimate_occupy_size:6291456, latest_update_ts:1727337630647485, estimated_finish_time:1727337647653148}) [2024-09-26 16:00:30.653143] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=11] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847408}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.653152] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272929d00, type:15, status:2, dag:{ObIDag:{this:0x7f9272929940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F3-0-0, dag_ret:0, dag_status:2, start_time:1727337630647445, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.653157] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=5] task finish process(ret=0, start_time=1727337630647482, end_time=1727337630653157, runtime=5675, *this={this:0x7f9272929d00, type:15, status:2, dag:{ObIDag:{this:0x7f9272929940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F3-0-0, dag_ret:0, dag_status:2, start_time:1727337630647445, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.653147] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:331}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:331}, table_count:1, [{i:0, table_key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}, merge_scn:{val:1726241465393532000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8633ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630652779, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=348us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.653162] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:1152921504606847408}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92728f3480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847408}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630638678, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=14.46ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.653172] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F5-0-0, dag_ret:0, dag_status:2, start_time:1727337630652587, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab480, output_block_cnt_arr:0x7f92684ab488, concurrent_cnt:1, estimate_row_cnt:45, estimate_occupy_size:0, latest_update_ts:1727337630652779, estimated_finish_time:1727337647653172}) [2024-09-26 16:00:30.653178] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292a760, type:15, status:2, dag:{ObIDag:{this:0x7f927292a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F5-0-0, dag_ret:0, dag_status:2, start_time:1727337630652587, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:30.653179] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337630621525, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.653183] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=5] task finish process(ret=0, start_time=1727337630652776, end_time=1727337630653183, runtime=407, *this={this:0x7f927292a760, type:15, status:2, dag:{ObIDag:{this:0x7f927292a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F5-0-0, dag_ret:0, dag_status:2, start_time:1727337630652587, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:30.653188] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752918752, storage_schema:{this:0x7f92684e9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828752918752, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.653197] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.653200] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.653204] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684e9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:309}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752918752, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.653221] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:309}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752918752, storage_schema:{this:0x7f92684e9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:309}, table_count:1, [{i:0, table_key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}, merge_scn:{val:1726241316899724000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8631da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630652545, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=657us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.653239] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272929e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F4-0-0, dag_ret:0, dag_status:2, start_time:1727337630652321, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}, scanned_row_cnt_arr:0x7f92684e9490, output_block_cnt_arr:0x7f92684e9498, concurrent_cnt:1, estimate_row_cnt:6, estimate_occupy_size:6291456, latest_update_ts:1727337630652545, estimated_finish_time:1727337647653242}) [2024-09-26 16:00:30.653244] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f927292a230, type:15, status:2, dag:{ObIDag:{this:0x7f9272929e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F4-0-0, dag_ret:0, dag_status:2, start_time:1727337630652321, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:30.653249] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=5] task finish process(ret=0, start_time=1727337630652543, end_time=1727337630653249, runtime=706, *this={this:0x7f927292a230, type:15, status:2, dag:{ObIDag:{this:0x7f9272929e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F4-0-0, dag_ret:0, dag_status:2, start_time:1727337630652321, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:30.653262] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337630620881, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.653271] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=8] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:30.653270] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272927f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EE-0-0, dag_ret:0, dag_status:2, start_time:1727337630638523, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f35d0, output_block_cnt_arr:0x7f92728f35d8, concurrent_cnt:1, estimate_row_cnt:35098, estimate_occupy_size:18874368, latest_update_ts:1727337630638678, estimated_finish_time:1727337647653279}) [2024-09-26 16:00:30.653276] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=5] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9268515480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.653279] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9272928310, type:15, status:2, dag:{ObIDag:{this:0x7f9272927f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EE-0-0, dag_ret:0, dag_status:2, start_time:1727337630638523, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.653284] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=4] task finish process(ret=0, start_time=1727337630638676, end_time=1727337630653284, runtime=14608, *this={this:0x7f9272928310, type:15, status:2, dag:{ObIDag:{this:0x7f9272927f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EE-0-0, dag_ret:0, dag_status:2, start_time:1727337630638523, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.653303] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337630621559, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.653314] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753357664, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753357664, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.653369] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=7] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:30.653372] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.653377] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:458}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753357664, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.653388] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=6] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3223, total_rows=22561) [2024-09-26 16:00:30.653391] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.653396] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847409}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.653394] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:458}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753357664, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:458}, table_count:1, [{i:0, table_key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8627b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630652798, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=577us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.653410] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:1152921504606847409}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9268515480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847409}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8640e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630644385, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=9.01ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.653512] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=15] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:30.653537] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272928480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EF-0-0, dag_ret:0, dag_status:2, start_time:1727337630641264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685155d0, output_block_cnt_arr:0x7f92685155d8, concurrent_cnt:1, estimate_row_cnt:32439, estimate_occupy_size:18874368, latest_update_ts:1727337630644385, estimated_finish_time:1727337647653546}) [2024-09-26 16:00:30.653546] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9272928840, type:15, status:2, dag:{ObIDag:{this:0x7f9272928480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EF-0-0, dag_ret:0, dag_status:2, start_time:1727337630641264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.653551] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=5] task finish process(ret=0, start_time=1727337630644383, end_time=1727337630653551, runtime=9168, *this={this:0x7f9272928840, type:15, status:2, dag:{ObIDag:{this:0x7f9272928480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EF-0-0, dag_ret:0, dag_status:2, start_time:1727337630641264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.653565] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=2] schedule one task(task=0x7f9272934ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.653570] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=4] schedule one task(task=0x7f926652a3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.653575] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=2] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.653580] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=2] schedule one task(task=0x7f9262cbe030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.653583] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=2] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.653627] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=1152921504606847408, range=0x7f92aaad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b56060, count:5, allocator:0x7f9275d767d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:30.653639] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=17] multi version scan iterator init succ(param.table_id_=1152921504606847409, range=0x7f92aa952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684e0060, count:5, allocator:0x7f92c19447d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684e0360, local_datums:0x7f92684e0360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:30.653685] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F6-0-0, dag_ret:0, dag_status:2, start_time:1727337630652777, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685e74a0, output_block_cnt_arr:0x7f92685e74a8, concurrent_cnt:1, estimate_row_cnt:14598, estimate_occupy_size:4194304, latest_update_ts:1727337630652798, estimated_finish_time:1727337647653687}) [2024-09-26 16:00:30.653695] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=10] succeed to init merge ctx(task={this:0x7f927292ac90, type:15, status:2, dag:{ObIDag:{this:0x7f927292a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F6-0-0, dag_ret:0, dag_status:2, start_time:1727337630652777, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.653700] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=5] task finish process(ret=0, start_time=1727337630652796, end_time=1727337630653700, runtime=904, *this={this:0x7f927292ac90, type:15, status:2, dag:{ObIDag:{this:0x7f927292a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F6-0-0, dag_ret:0, dag_status:2, start_time:1727337630652777, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.653764] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=7] multi version scan iterator init succ(param.table_id_=331, range=0x7f92ab152200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a5e060, count:5, allocator:0x7f92c195e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9264a5e360, local_datums:0x7f9264a5e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:30.653874] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=22][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:30.653887] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:30.653911] ERROR issue_dba_error (ob_log.cpp:1875) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.653914] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.653919] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=5][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:30.653921] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:30.653925] ERROR issue_dba_error (ob_log.cpp:1875) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.653923] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 7, flag: 0, null: 0, ptr: 0x7f92665aa178, hex: 31303033343330, cstr: 1003430},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f92665aa17f, hex: 3230323430393133303435303537, cstr: 20240913045057},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa0e0, hex: 6E48500300000000, int: 55593070},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa118, hex: 58B03BC5A1610BE8, int: -1726178684739801000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-26357, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:250, row_store_type:0, opt:5, var_column_count:0, row_offset:15394, original_length:16334, max_merged_trans_version:1726190643638412000, data_length:16334, data_zlength:5786, data_checksum:2846467531, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9268560030, buf_size:5786, data_size:16334, row_count:250, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:796410710, row_count_delta:250, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16334}) [2024-09-26 16:00:30.653928] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.653934] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=10][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:30.653935] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:30.653935] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:30.653937] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:30.653937] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:30.653938] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:30.653938] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9263b5e178, hex: 30323232362D32312D373825, cstr: 02226-21-78%},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9263b5e184, hex: 3230323430393133303435343039, cstr: 20240913045409},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e0e0, hex: CD7F500300000000, int: 55607245},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e118, hex: E8EC05CCA0600BE8, int: -1726179788432479000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-11869, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:238, row_store_type:0, opt:5, var_column_count:0, row_offset:15429, original_length:16321, max_merged_trans_version:1726190643638412000, data_length:16321, data_zlength:5388, data_checksum:152456761, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9263bde030, buf_size:5388, data_size:16321, row_count:238, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2114797869, row_count_delta:238, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16321}) [2024-09-26 16:00:30.653950] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=12][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:30.653952] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:30.653953] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:30.653955] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:30.653973] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:30.653940] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684e0360, local_datums:0x7f92684e0360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e03e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e0418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e0450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684e0360, local_datums:0x7f92684e0360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e03e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e0418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e0450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92c1944118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c1944118, limit_param:null, stmt_allocator:0x7f92c19447d8, allocator:0x7f92c19447d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:9114723692158910464, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684e0360, local_datums:0x7f92684e0360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e03e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e0418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e0450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337630620881, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:30.653987] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=47][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:30.653956] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9275d76118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9275d76118, limit_param:null, stmt_allocator:0x7f9275d767d8, allocator:0x7f9275d767d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337630620851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:30.654005] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=49][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:30.654012] ERROR issue_dba_error (ob_log.cpp:1875) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.653988] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684e0360, local_datums:0x7f92684e0360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e03e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e0418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e0450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92c1944118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c1944118, limit_param:null, stmt_allocator:0x7f92c19447d8, allocator:0x7f92c19447d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:9114723692158910464, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684e0360, local_datums:0x7f92684e0360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e03e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e0418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e0450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337630620881, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:30.654019] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=31][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:30.654015] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.654023] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:30.654025] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:30.654021] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=2] partition merge iter row count(i=0, row_count=250, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337630620881, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.654032] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:30.654027] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e070, hex: 0000000000000000, int: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e0a8, hex: 3501000000000000, int: 309},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e0e0, hex: 3501000000000000, int: 309},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e118, hex: 7032CD318A310BE8, int: -1726231562555674000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-24587, column_count:14, rowkey_column_count:5, has_column_checksum:0, row_count:196, row_store_type:0, opt:0, var_column_count:0, row_offset:15642, original_length:16366, max_merged_trans_version:1726241465393532000, data_length:16366, data_zlength:16366, data_checksum:3184742917, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6604090, buf_size:16366, data_size:16366, row_count:196, column_count:14, max_merged_trans_version:1726241465393532000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2306482841, row_count_delta:7, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16366}) [2024-09-26 16:00:30.654007] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9275d76118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9275d76118, limit_param:null, stmt_allocator:0x7f9275d767d8, allocator:0x7f9275d767d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337630620851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:30.654038] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=10][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:30.654038] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=31][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:30.654039] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=1][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:30.654041] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:30.654042] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:30.654042] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:30.654044] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:30.654045] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:30.654045] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:30.654040] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=1] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337630620851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.654051] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:30.654046] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337630653594, end_time=1727337630654046, runtime=452, *this={this:0x7f926652a3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9272928480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EF-0-0, dag_ret:0, dag_status:2, start_time:1727337630641264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.654053] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=7][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.654059] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=1][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.654071] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337630653593, end_time=1727337630654071, runtime=478, *this={this:0x7f9272934ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9272927f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EE-0-0, dag_ret:0, dag_status:2, start_time:1727337630638523, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.654072] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=15] dag finished(dag_ret=-4184, runtime=12808, dag_cnt=36, dag_cnts_[dag.get_type()]=36, &dag=0x7f9272928480, dag={ObIDag:{this:0x7f9272928480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EF-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630641264, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.654082] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EF-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337630641256, task_id:YB42C0A8002D-0006230056D8C5EF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.654047] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9264a5e360, local_datums:0x7f9264a5e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e3a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e3e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e4f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e5a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e5d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:331}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752994696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9264a5e360, local_datums:0x7f9264a5e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e3a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e3e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e4f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e5a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e5d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, store_ctx:{this:0x7f92c195e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c195e118, limit_param:null, stmt_allocator:0x7f92c195e7d8, allocator:0x7f92c195e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241465393532000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:-1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9264a5e360, local_datums:0x7f9264a5e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e3a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e3e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e4f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e5a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e5d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, key:{"BIGINT":0},{"BIGINT":309},{"BIGINT":309}, value_iter:{value:{this=0x7f92830f5dd8 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=15 list_head=0x7f928848eeb8 latest_compact_node=0x7f9283113eb0 max_trans_version={val:1726241465393532000, v:0} max_trans_id=125582299 max_elr_trans_version={val:1726241465393532000, v:0} max_elr_trans_id=125582299 latest_compact_ts=1727334207881700 last_compact_cnt=49 total_trans_node_cnt=141 max_modify_scn={val:1726241465393532000, v:0} min_modify_scn={val:1726179340987317000, v:0}}, version_iter:this=0x7f928848eb78 trans_version={val:1726241465393532000, v:0} scn={val:1726241465393532000, v:0} tx_id={txid:125582299} prev=0x7f928848e850 next=0x7f928848eeb8 modify_count=65 acc_checksum=2418483105 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:42} seq_no=1726241465159151, multi_version_iter:this=0x7f9281a21200 trans_version={val:1726230662165479000, v:0} scn={val:1726230662165479000, v:0} tx_id={txid:125497658} prev=0x7f9281a20ec0 next=0x7f9283111860 modify_count=41 acc_checksum=883883390 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:76} seq_no=1726230662033825, max_committed_trans_version:1726241465393532000, cur_trans_version:{val:1726231562555674000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241465393532000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:196, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337630621542, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:14, cur_pos:0, count:0}}) [2024-09-26 16:00:30.654126] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=79][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.654129] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=0] partition merge iter row count(i=0, row_count=196, ghost_row_count=0, pkey={tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, table={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337630621542, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.654149] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630653722, end_time=1727337630654148, runtime=426, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f927292a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F5-0-0, dag_ret:0, dag_status:2, start_time:1727337630652587, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:30.654154] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=15630, dag_cnt=35, dag_cnts_[dag.get_type()]=35, &dag=0x7f9272927f50, dag={ObIDag:{this:0x7f9272927f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5EE-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630638523, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.654172] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5EE-0-0] [lt=15] succeed to del sys task(removed_task={start_time:1727337630638512, task_id:YB42C0A8002D-0006230056D8C5EE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.654194] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=1] schedule one task(task=0x7f9262cee030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.654201] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337630654200, task_id:YB42C0A8002D-0006230056D8C5F7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.654204] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=3] schedule one task(task=0x7f927292b1c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.654209] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630654208, task_id:YB42C0A8002D-0006230056D8C5F8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.654211] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=2] schedule one task(task=0x7f927292b6f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.654206] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=309, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:11}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92728fa060, count:7, allocator:0x7f926363e7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92728fa3b0, local_datums:0x7f92728fa3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:30.654256] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=458, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a60060, count:8, allocator:0x7f926360a7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264a603e0, local_datums:0x7f9264a603e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:30.654372] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=21][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.654405] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.654408] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.654418] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=0] partition merge iter row count(i=0, row_count=156, ghost_row_count=0, pkey={tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, table={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337630621525, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.654447] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630654167, end_time=1727337630654446, runtime=279, *this={this:0x7f9262cbe030, type:1, status:2, dag:{ObIDag:{this:0x7f9272929e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F4-0-0, dag_ret:0, dag_status:2, start_time:1727337630652321, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:30.654457] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2136, dag_cnt=34, dag_cnts_[dag.get_type()]=34, &dag=0x7f9272929e70, dag={ObIDag:{this:0x7f9272929e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F4-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630652321, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}) [2024-09-26 16:00:30.654464] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F4-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630652317, task_id:YB42C0A8002D-0006230056D8C5F4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.654510] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.654546] ERROR issue_dba_error (ob_log.cpp:1875) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.654549] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.654544] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=444, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c70060, count:6, allocator:0x7f9269f507d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9262c70390, local_datums:0x7f9262c70390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:30.654559] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=0] partition merge iter row count(i=0, row_count=247, ghost_row_count=0, pkey={tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337630621559, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.654580] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630654222, end_time=1727337630654579, runtime=357, *this={this:0x7f9262cee030, type:1, status:2, dag:{ObIDag:{this:0x7f927292a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F6-0-0, dag_ret:0, dag_status:2, start_time:1727337630652777, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.654589] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1812, dag_cnt=33, dag_cnts_[dag.get_type()]=33, &dag=0x7f927292a8d0, dag={ObIDag:{this:0x7f927292a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F6-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630652777, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.654596] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F6-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337630652772, task_id:YB42C0A8002D-0006230056D8C5F6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.654626] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.654629] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.654631] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.654637] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.654640] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861cf80, is_inited:true, wash_score:1727337630620660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:528, tablet_addr:[816017-1849-0-2305843670640488448-2](offset=1830912,size=616,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:455}, data_tablet_id:{id:455}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4242650745, row_count:15909}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861d6ec, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305845137371811840-0](offset=1822720,size=1982,type=2,seq=0), type:"MAJOR", tablet_id:{id:455}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389074974000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305845058988660670-0](offset=1824702,size=1909,type=2,seq=0), type:"MINOR", tablet_id:{id:455}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178192070487000, upper_trans_version:1726178192070487000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337630621594, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcec60, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209461810}, this:0x7f92a3bcec60, timestamp:1727334209461810, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:55, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274239678928000, v:0}, rec_scn:{val:1726241818022255001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771548160-2](offset=1826816,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843202489049421-0](offset=1827149,size=180,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881173505-0](offset=1827329,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447693539-0](offset=1827555,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a794f0, 0x7f92a3bcec60, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.654683] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=42] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.654690] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.654706] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630654704, task_id:YB42C0A8002D-0006230056D8C5F9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.654709] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=3] schedule one task(task=0x7f927292bc20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.654722] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630654721, task_id:YB42C0A8002D-0006230056D8C5FA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.654724] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=2] schedule one task(task=0x7f927292c150, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.654726] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.654737] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.654741] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.654740] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=19][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.654743] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.654743] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.654745] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.654747] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.654747] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.654750] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.654751] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.654754] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.654758] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.654758] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.654761] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.654770] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=0] partition merge iter row count(i=0, row_count=207, ghost_row_count=0, pkey={tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337630621505, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.654788] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630654511, end_time=1727337630654788, runtime=277, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272929940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F3-0-0, dag_ret:0, dag_status:2, start_time:1727337630647445, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.654796] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=7351, dag_cnt=32, dag_cnts_[dag.get_type()]=32, &dag=0x7f9272929940, dag={ObIDag:{this:0x7f9272929940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F3-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630647445, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.654803] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F3-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630647438, task_id:YB42C0A8002D-0006230056D8C5F3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.654831] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.654834] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.654836] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.654839] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.654841] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b863cf80, is_inited:true, wash_score:1727337630620660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:522, tablet_addr:[816017-1849-0-2305843664198197248-2](offset=1990656,size=610,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:487}, data_tablet_id:{id:487}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1549633270, row_count:11693}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863d6c8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844647745695744-0](offset=1978368,size=1526,type=2,seq=0), type:"MAJOR", tablet_id:{id:487}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389060348000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843977730799094-0](offset=1979894,size=902,type=2,seq=0), type:"MINOR", tablet_id:{id:487}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178190844721000, upper_trans_version:1726178190844721000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337630621576, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd4850, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209506557}, this:0x7f92a3bd4850, timestamp:1727334209506557, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:66, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274231198751000, v:0}, rec_scn:{val:1726241813664082000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843367845449728-2](offset=1986560,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046758222-0](offset=1986894,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881333244-0](offset=1987068,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447853278-0](offset=1987294,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a79cc0, 0x7f92a3bd4850, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.654876] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=35] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.654881] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.654894] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:30.654915] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630654914, task_id:YB42C0A8002D-0006230056D8C5FB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.654918] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=2] schedule one task(task=0x7f927292c680, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.654925] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2338, dag_cnt=31, dag_cnts_[dag.get_type()]=31, &dag=0x7f927292a3a0, dag={ObIDag:{this:0x7f927292a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F5-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630652587, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}) [2024-09-26 16:00:30.654934] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F5-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337630652582, task_id:YB42C0A8002D-0006230056D8C5F5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.654936] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.654939] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.654941] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.654945] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.654949] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630654948, task_id:YB42C0A8002D-0006230056D8C5FC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.654949] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.654952] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=2] schedule one task(task=0x7f927292cbb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.654955] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.654965] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.654967] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.654969] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726077706339704027) [2024-09-26 16:00:30.654972] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.654974] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.654978] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.655130] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337630621611, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.655131] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337630621594, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.655143] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753437040, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=57, max_schema_version_in_memtable=1710828753437040, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.655150] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753332736, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=36, max_schema_version_in_memtable=1710828753332736, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.655238] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=24] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1362, total_rows=1362) [2024-09-26 16:00:30.655243] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.655249] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:492}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:12, rowkey_column_count:11, schema_rowkey_col_cnt:9, full_stored_col_cnt:59, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753437040, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=28 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.655276] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=20] succeed to build merge ctx(tablet_id={id:492}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753437040, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:492}, table_count:1, [{i:0, table_key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8624c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630654732, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=514us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.655364] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F9-0-0, dag_ret:0, dag_status:2, start_time:1727337630654709, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b234c0, output_block_cnt_arr:0x7f9264b234c8, concurrent_cnt:1, estimate_row_cnt:1362, estimate_occupy_size:2097152, latest_update_ts:1727337630654732, estimated_finish_time:1727337647655365}) [2024-09-26 16:00:30.655372] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f927292bc20, type:15, status:2, dag:{ObIDag:{this:0x7f927292b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F9-0-0, dag_ret:0, dag_status:2, start_time:1727337630654709, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.655377] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=5] task finish process(ret=0, start_time=1727337630654730, end_time=1727337630655377, runtime=647, *this={this:0x7f927292bc20, type:15, status:2, dag:{ObIDag:{this:0x7f927292b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F9-0-0, dag_ret:0, dag_status:2, start_time:1727337630654709, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.655410] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=35] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:30.655415] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.655428] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337630621659, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.655422] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:455}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:10, rowkey_column_count:9, schema_rowkey_col_cnt:7, full_stored_col_cnt:38, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753332736, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=26 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.655439] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753360568, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753360568, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.655451] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.655455] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.655446] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=18] succeed to build merge ctx(tablet_id={id:455}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753332736, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:455}, table_count:1, [{i:0, table_key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630654620, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=800us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.655460] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:472}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753360568, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.655490] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=25] succeed to build merge ctx(tablet_id={id:472}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753360568, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:472}, table_count:1, [{i:0, table_key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630654930, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=528us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.655511] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FB-0-0, dag_ret:0, dag_status:2, start_time:1727337630654918, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d460, output_block_cnt_arr:0x7f9264b1d468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337630654930, estimated_finish_time:1727337647655511}) [2024-09-26 16:00:30.655517] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292c680, type:15, status:2, dag:{ObIDag:{this:0x7f927292c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FB-0-0, dag_ret:0, dag_status:2, start_time:1727337630654918, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.655522] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=5] task finish process(ret=0, start_time=1727337630654928, end_time=1727337630655522, runtime=594, *this={this:0x7f927292c680, type:15, status:2, dag:{ObIDag:{this:0x7f927292c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FB-0-0, dag_ret:0, dag_status:2, start_time:1727337630654918, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.655610] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=36] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:30.655772] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F8-0-0, dag_ret:0, dag_status:2, start_time:1727337630654211, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685e74b0, output_block_cnt_arr:0x7f92685e74b8, concurrent_cnt:1, estimate_row_cnt:3824, estimate_occupy_size:4194304, latest_update_ts:1727337630654620, estimated_finish_time:1727337647655774}) [2024-09-26 16:00:30.655783] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=10] succeed to init merge ctx(task={this:0x7f927292b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f927292b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F8-0-0, dag_ret:0, dag_status:2, start_time:1727337630654211, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.655788] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=5] task finish process(ret=0, start_time=1727337630654618, end_time=1727337630655788, runtime=1170, *this={this:0x7f927292b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f927292b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F8-0-0, dag_ret:0, dag_status:2, start_time:1727337630654211, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.655821] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337630621680, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.655832] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753001832, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=21, max_schema_version_in_memtable=1710828753001832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.655844] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.655846] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.655851] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:342}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:23, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753001832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.655872] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:342}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753001832, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:342}, table_count:1, [{i:0, table_key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630654961, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=888us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.655892] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FC-0-0, dag_ret:0, dag_status:2, start_time:1727337630654951, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f490, output_block_cnt_arr:0x7f9264b1f498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337630654961, estimated_finish_time:1727337647655891}) [2024-09-26 16:00:30.655897] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f927292cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f927292c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FC-0-0, dag_ret:0, dag_status:2, start_time:1727337630654951, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.655901] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=4] task finish process(ret=0, start_time=1727337630654960, end_time=1727337630655901, runtime=941, *this={this:0x7f927292cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f927292c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FC-0-0, dag_ret:0, dag_status:2, start_time:1727337630654951, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.655917] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337630621628, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.655925] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753441792, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828753441792, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.655975] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=6] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1295, total_rows=1295) [2024-09-26 16:00:30.655977] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.655981] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:501}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753441792, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.656010] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=24] succeed to build merge ctx(tablet_id={id:501}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753441792, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:501}, table_count:1, [{i:0, table_key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630654737, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.24ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.656089] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=23] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FA-0-0, dag_ret:0, dag_status:2, start_time:1727337630654724, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1490, output_block_cnt_arr:0x7f92728f1498, concurrent_cnt:1, estimate_row_cnt:1295, estimate_occupy_size:0, latest_update_ts:1727337630654737, estimated_finish_time:1727337647656089}) [2024-09-26 16:00:30.656095] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292c150, type:15, status:2, dag:{ObIDag:{this:0x7f927292bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FA-0-0, dag_ret:0, dag_status:2, start_time:1727337630654724, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.656100] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=4] task finish process(ret=0, start_time=1727337630654734, end_time=1727337630656099, runtime=1365, *this={this:0x7f927292c150, type:15, status:2, dag:{ObIDag:{this:0x7f927292bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FA-0-0, dag_ret:0, dag_status:2, start_time:1727337630654724, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.656119] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337630621576, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.656127] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753417144, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828753417144, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.656200] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:30.656203] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.656207] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:487}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753417144, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.656224] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:487}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753417144, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:487}, table_count:1, [{i:0, table_key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630654827, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.38ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.656301] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=14] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:30.656398] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=2] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F7-0-0, dag_ret:0, dag_status:2, start_time:1727337630654204, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f34a0, output_block_cnt_arr:0x7f92728f34a8, concurrent_cnt:1, estimate_row_cnt:2199, estimate_occupy_size:0, latest_update_ts:1727337630654827, estimated_finish_time:1727337647656398}) [2024-09-26 16:00:30.656405] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f927292b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f927292ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F7-0-0, dag_ret:0, dag_status:2, start_time:1727337630654204, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.656410] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=4] task finish process(ret=0, start_time=1727337630654825, end_time=1727337630656410, runtime=1585, *this={this:0x7f927292b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f927292ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F7-0-0, dag_ret:0, dag_status:2, start_time:1727337630654204, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.656444] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=1] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.656450] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=4] schedule one task(task=0x7f9265aa63c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.656469] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=1] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.656472] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=2] schedule one task(task=0x7f9264ae83c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.656475] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=2] schedule one task(task=0x7f9264ae8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.656478] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=2] schedule one task(task=0x7f9264b36030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.656523] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=492, range=0x7f92aae56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:57, schema_rowkey_cnt:9, rowkey_cnt:11, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:9, column_cnt:59}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d60060, count:11, allocator:0x7f92636067d8, capacity:11, init_cnt:11}, datum_utils:{is_oracle_mode:false, rowkey_cnt:11, col_cnt:11, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:59, datum_buffer:{capacity:59, datums:0x7f926589a050, local_datums:0x7f9264d60440}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,col_id=43:NOP,col_id=44:NOP,col_id=45:NOP,col_id=46:NOP,col_id=47:NOP,col_id=48:NOP,col_id=49:NOP,col_id=50:NOP,col_id=51:NOP,col_id=52:NOP,col_id=53:NOP,col_id=54:NOP,col_id=55:NOP,col_id=56:NOP,col_id=57:NOP,col_id=58:NOP,]}) [2024-09-26 16:00:30.656533] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=487, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:11}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f926589c060, count:8, allocator:0x7f92691327d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f926589c3e0, local_datums:0x7f926589c3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:30.656735] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=19][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.656749] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.656775] ERROR issue_dba_error (ob_log.cpp:1875) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.656778] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.656787] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.656790] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.656788] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=0] partition merge iter row count(i=0, row_count=61, ghost_row_count=0, pkey={tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337630621611, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.656800] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=0] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337630621576, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.656808] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630656480, end_time=1727337630656808, runtime=328, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f927292b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F9-0-0, dag_ret:0, dag_status:2, start_time:1727337630654709, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.656817] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2108, dag_cnt=30, dag_cnts_[dag.get_type()]=30, &dag=0x7f927292b860, dag={ObIDag:{this:0x7f927292b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F9-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630654709, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.656820] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630656490, end_time=1727337630656819, runtime=329, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f927292ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F7-0-0, dag_ret:0, dag_status:2, start_time:1727337630654204, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.656825] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F9-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630654704, task_id:YB42C0A8002D-0006230056D8C5F9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.656886] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=342, range=0x7f92a9b52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:21, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:23}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dce060, count:6, allocator:0x7f9269ffa7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:23, datum_buffer:{capacity:32, datums:0x7f9287dce390, local_datums:0x7f9287dce390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,]}) [2024-09-26 16:00:30.656896] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=455, range=0x7f92a98d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:36, schema_rowkey_cnt:7, rowkey_cnt:9, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:7, column_cnt:38}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dda060, count:9, allocator:0x7f92a31767d8, capacity:9, init_cnt:9}, datum_utils:{is_oracle_mode:false, rowkey_cnt:9, col_cnt:9, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:38, datum_buffer:{capacity:38, datums:0x7f9287ddb340, local_datums:0x7f9287dda3f0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,]}) [2024-09-26 16:00:30.656913] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.656940] ERROR issue_dba_error (ob_log.cpp:1875) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.656942] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.656950] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337630621680, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.656967] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630656853, end_time=1727337630656967, runtime=114, *this={this:0x7f9264b36030, type:1, status:2, dag:{ObIDag:{this:0x7f927292c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FC-0-0, dag_ret:0, dag_status:2, start_time:1727337630654951, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.656975] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2024, dag_cnt=29, dag_cnts_[dag.get_type()]=29, &dag=0x7f927292c7f0, dag={ObIDag:{this:0x7f927292c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FC-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630654951, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.656981] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FC-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630654948, task_id:YB42C0A8002D-0006230056D8C5FC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.657161] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.657207] ERROR issue_dba_error (ob_log.cpp:1875) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.657210] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.657219] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=0] partition merge iter row count(i=0, row_count=140, ghost_row_count=0, pkey={tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337630621594, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.657239] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630656836, end_time=1727337630657238, runtime=402, *this={this:0x7f9265aa63c0, type:1, status:2, dag:{ObIDag:{this:0x7f927292b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F8-0-0, dag_ret:0, dag_status:2, start_time:1727337630654211, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.657247] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3036, dag_cnt=28, dag_cnts_[dag.get_type()]=28, &dag=0x7f927292b330, dag={ObIDag:{this:0x7f927292b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F8-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630654211, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.657255] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F8-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630654208, task_id:YB42C0A8002D-0006230056D8C5F8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.657317] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=7] multi version scan iterator init succ(param.table_id_=501, range=0x7f92aac56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a64060, count:6, allocator:0x7f926366c7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264a64390, local_datums:0x7f9264a64390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:30.657389] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.657416] ERROR issue_dba_error (ob_log.cpp:1875) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.657418] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.657426] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=0] partition merge iter row count(i=0, row_count=56, ghost_row_count=0, pkey={tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337630621628, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.657440] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630657290, end_time=1727337630657440, runtime=150, *this={this:0x7f9264ae83c0, type:1, status:2, dag:{ObIDag:{this:0x7f927292bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FA-0-0, dag_ret:0, dag_status:2, start_time:1727337630654724, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.657447] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2723, dag_cnt=27, dag_cnts_[dag.get_type()]=27, &dag=0x7f927292bd90, dag={ObIDag:{this:0x7f927292bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FA-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630654724, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.657453] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FA-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630654721, task_id:YB42C0A8002D-0006230056D8C5FA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.657505] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=472, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d62060, count:3, allocator:0x7f9269f507d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f9264d62320, local_datums:0x7f9264d62320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:30.657537] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=10][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.657563] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.657565] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.657573] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=0] partition merge iter row count(i=0, row_count=17, ghost_row_count=0, pkey={tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337630621659, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.657589] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630657473, end_time=1727337630657589, runtime=116, *this={this:0x7f9264ae8030, type:1, status:2, dag:{ObIDag:{this:0x7f927292c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FB-0-0, dag_ret:0, dag_status:2, start_time:1727337630654918, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.657596] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2678, dag_cnt=26, dag_cnts_[dag.get_type()]=26, &dag=0x7f927292c2c0, dag={ObIDag:{this:0x7f927292c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FB-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630654918, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.657602] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FB-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630654914, task_id:YB42C0A8002D-0006230056D8C5FB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.657637] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3433, dag_cnt=25, dag_cnts_[dag.get_type()]=25, &dag=0x7f927292ae00, dag={ObIDag:{this:0x7f927292ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5F7-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630654204, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.657642] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5F7-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337630654200, task_id:YB42C0A8002D-0006230056D8C5F7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.657657] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337630657655, task_id:YB42C0A8002D-0006230056D8C5FD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.657662] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=4] schedule one task(task=0x7f927292d0e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.657669] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630657668, task_id:YB42C0A8002D-0006230056D8C5FE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.657672] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=2] schedule one task(task=0x7f927292d610, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.657677] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630657676, task_id:YB42C0A8002D-0006230056D8C5FF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.657679] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=2] schedule one task(task=0x7f927292db40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.657682] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630657682, task_id:YB42C0A8002D-0006230056D8C600-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.657684] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=2] schedule one task(task=0x7f927292e070, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.657688] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630657687, task_id:YB42C0A8002D-0006230056D8C601-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.657690] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=1] schedule one task(task=0x7f927292e5a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.657694] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337630657693, task_id:YB42C0A8002D-0006230056D8C602-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.657698] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=4] schedule one task(task=0x7f927292ead0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.657705] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.657708] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.657710] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:30.657712] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=18][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.657714] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.657716] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.657718] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.657722] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.657725] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.657732] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.657754] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.657717] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726112893187579000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864b040, is_inited:true, wash_score:1727337630620660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[812168-2208-0-2305843644869742592-2](offset=888832,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:323}, data_tablet_id:{id:323}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726112893187579000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2130074741, row_count:1002}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864b71c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[812168-2208-0-2305844507084419072-0](offset=880640,size=1395,type=2,seq=0), type:"MAJOR", tablet_id:{id:323}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562725180000}}, minor_tables:{{i:0, addr:[812168-2208-0-2305843933706286451-0](offset=882035,size=861,type=2,seq=0), type:"MINI", tablet_id:{id:323}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726112893187579000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726112601715820000, upper_trans_version:1726112601715820000}, {i:1, addr:[812168-2208-0-2305843933706287312-0](offset=882896,size=861,type=2,seq=0), type:"MINI", tablet_id:{id:323}, scn_range:{start_scn:{val:1726112893187579000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149985954467000, upper_trans_version:1726149985954467000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337630621751, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd9e40, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220581091}, this:0x7f92a3bd9e40, timestamp:1727334220581091, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:77, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265243868557000, v:0}, rec_scn:{val:1726264808086409001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[812168-2208-0-2305843482734723072-2](offset=884736,size=441,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[812168-2208-0-2305843176718303673-0](offset=885177,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843251880231509-0](offset=885333,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843216446751543-0](offset=885559,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7d370, 0x7f92a3bd9e40, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.657757] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.657759] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=42] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.657761] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.657763] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.657765] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.657765] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8641da0, is_inited:true, wash_score:1727337630620660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:487, tablet_addr:[816017-1849-0-2305843626617249792-2](offset=2007040,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101009}, data_tablet_id:{id:140}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726073917016589000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3251381279, row_count:150}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8642458, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844188184215552-0](offset=1998848,size=1098,type=2,seq=0), type:"MAJOR", tablet_id:{id:101009}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726070403667750001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843957329724490-0](offset=1999946,size=883,type=2,seq=0), type:"MINOR", tablet_id:{id:101009}, scn_range:{start_scn:{val:1726073917016589000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726167395874925000, upper_trans_version:1726167395874925000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337630621717, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd0ba0, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495816}, this:0x7f92a3bd0ba0, timestamp:1727334209495816, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:59, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726256203821911000, v:0}, rec_scn:{val:1726241803398655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697982464-2](offset=2002944,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276970828-0](offset=2003276,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881349602-0](offset=2003426,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7bc00, 0x7f92a3bd0ba0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.657815] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=50] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.657821] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.657837] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.657839] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.657841] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:30.657844] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.657853] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=8] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.657856] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.657902] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.657905] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.657906] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:30.657909] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.657912] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864df20, is_inited:true, wash_score:1727337630620660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:503, tablet_addr:[814042-5586-0-2305843643796283392-2](offset=1171456,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:377}, data_tablet_id:{id:377}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4164021307, row_count:17}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864e5fc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844760487772160-0](offset=1163264,size=1631,type=2,seq=0), type:"MAJOR", tablet_id:{id:377}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563080818000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843989541144159-0](offset=1164895,size=913,type=2,seq=0), type:"MINI", tablet_id:{id:377}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557779816000, upper_trans_version:1726121557779816000}, {i:1, addr:[814042-5586-0-2305843969140050416-0](offset=1165808,size=894,type=2,seq=0), type:"MINI", tablet_id:{id:377}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986451466000, upper_trans_version:1726149986451466000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337630621769, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdb5b0, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220773832}, this:0x7f92a3bdb5b0, timestamp:1727334220773832, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:80, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293334630000, v:0}, rec_scn:{val:1726264800181268000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513780224-2](offset=1167360,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843176718586294-0](offset=1167798,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880514130-0](offset=1167954,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447034164-0](offset=1168180,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7e310, 0x7f92a3bdb5b0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.657944] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=32] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.657948] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.657966] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=8][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.657982] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=15][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.657984] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:30.657987] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.657991] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.657995] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.658024] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337630621700, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.658034] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.658045] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.658048] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.658053] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:140}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.658069] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:140}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:140}, table_count:1, [{i:0, table_key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8622d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630657691, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=360us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.658091] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FD-0-0, dag_ret:0, dag_status:2, start_time:1727337630657661, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505460, output_block_cnt_arr:0x7f9266505468, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337630657691, estimated_finish_time:1727337647658091}) [2024-09-26 16:00:30.658097] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f927292cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FD-0-0, dag_ret:0, dag_status:2, start_time:1727337630657661, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.658103] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=5] task finish process(ret=0, start_time=1727337630657689, end_time=1727337630658102, runtime=413, *this={this:0x7f927292d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f927292cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FD-0-0, dag_ret:0, dag_status:2, start_time:1727337630657661, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.658150] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=2] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.658148] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337630621751, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.658153] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337630621785, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.658158] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752949376, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=20, max_schema_version_in_memtable=1710828752949376, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.658162] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=16, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.658172] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.658174] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.658179] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.658183] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.658178] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:297}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:18, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.658188] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:323}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:22, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752949376, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.658194] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:297}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:297}, table_count:1, [{i:0, table_key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630657834, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=343us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.658211] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C602-0-0, dag_ret:0, dag_status:2, start_time:1727337630657698, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5490, output_block_cnt_arr:0x7f9287db5498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337630657834, estimated_finish_time:1727337647658211}) [2024-09-26 16:00:30.658216] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f927292ead0, type:15, status:2, dag:{ObIDag:{this:0x7f927292e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C602-0-0, dag_ret:0, dag_status:2, start_time:1727337630657698, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.658206] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:323}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752949376, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:323}, table_count:1, [{i:0, table_key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630657700, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=486us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.658221] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=4] task finish process(ret=0, start_time=1727337630657833, end_time=1727337630658220, runtime=387, *this={this:0x7f927292ead0, type:15, status:2, dag:{ObIDag:{this:0x7f927292e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C602-0-0, dag_ret:0, dag_status:2, start_time:1727337630657698, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.658237] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C600-0-0, dag_ret:0, dag_status:2, start_time:1727337630657684, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a43480, output_block_cnt_arr:0x7f9263a43488, concurrent_cnt:1, estimate_row_cnt:205, estimate_occupy_size:0, latest_update_ts:1727337630657700, estimated_finish_time:1727337647658237}) [2024-09-26 16:00:30.658243] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292e070, type:15, status:2, dag:{ObIDag:{this:0x7f927292dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C600-0-0, dag_ret:0, dag_status:2, start_time:1727337630657684, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.658248] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=5] task finish process(ret=0, start_time=1727337630657694, end_time=1727337630658248, runtime=554, *this={this:0x7f927292e070, type:15, status:2, dag:{ObIDag:{this:0x7f927292dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C600-0-0, dag_ret:0, dag_status:2, start_time:1727337630657684, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.658255] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=140, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:3, allocator:0x7f926919e7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5e320, local_datums:0x7f9263b5e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:30.658307] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.658340] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.658343] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.658352] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337630621700, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.658368] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337630621717, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.658370] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630658172, end_time=1727337630658370, runtime=198, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f927292cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FD-0-0, dag_ret:0, dag_status:2, start_time:1727337630657661, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.658376] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.658379] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=718, dag_cnt=24, dag_cnts_[dag.get_type()]=24, &dag=0x7f927292cd20, dag={ObIDag:{this:0x7f927292cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FD-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630657661, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.658384] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.658387] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FD-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630657655, task_id:YB42C0A8002D-0006230056D8C5FD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.658397] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.658401] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101009}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:30.658422] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=4] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.658429] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=5] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.658446] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630658445, task_id:YB42C0A8002D-0006230056D8C603-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.658424] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=19] succeed to build merge ctx(tablet_id={id:101009}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101009}, table_count:1, [{i:0, table_key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8641da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630657751, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=649us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.658450] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=4] schedule one task(task=0x7f927292f000, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.658477] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=27] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FE-0-0, dag_ret:0, dag_status:2, start_time:1727337630657672, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3470, output_block_cnt_arr:0x7f9287db3478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337630657751, estimated_finish_time:1727337647658477}) [2024-09-26 16:00:30.658484] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292d610, type:15, status:2, dag:{ObIDag:{this:0x7f927292d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FE-0-0, dag_ret:0, dag_status:2, start_time:1727337630657672, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.658488] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=4] task finish process(ret=0, start_time=1727337630657749, end_time=1727337630658488, runtime=739, *this={this:0x7f927292d610, type:15, status:2, dag:{ObIDag:{this:0x7f927292d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FE-0-0, dag_ret:0, dag_status:2, start_time:1727337630657672, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.658511] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.658514] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.658516] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:30.658519] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.658522] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.658527] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.658519] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=297, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:16, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:18}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ec060, count:6, allocator:0x7f926364a7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:18, datum_buffer:{capacity:32, datums:0x7f92664ec390, local_datums:0x7f92664ec390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,]}) [2024-09-26 16:00:30.658554] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=21][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.658567] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=323, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:20, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:22}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a60060, count:5, allocator:0x7f9275dfe7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f9264a60360, local_datums:0x7f9264a60360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,]}) [2024-09-26 16:00:30.658586] ERROR issue_dba_error (ob_log.cpp:1875) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.658589] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.658597] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337630621785, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.658615] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630658478, end_time=1727337630658615, runtime=137, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f927292e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C602-0-0, dag_ret:0, dag_status:2, start_time:1727337630657698, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.658624] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=926, dag_cnt=23, dag_cnts_[dag.get_type()]=23, &dag=0x7f927292e710, dag={ObIDag:{this:0x7f927292e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C602-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630657698, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.658630] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C602-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630657693, task_id:YB42C0A8002D-0006230056D8C602-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.658666] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=2] schedule one task(task=0x7f9272934ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.658673] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337630658672, task_id:YB42C0A8002D-0006230056D8C604-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.658677] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=4] schedule one task(task=0x7f927292f530, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.658694] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.658697] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.658699] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:30.658704] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.658707] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.658715] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.658804] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.658843] ERROR issue_dba_error (ob_log.cpp:1875) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.658846] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.658856] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=0] partition merge iter row count(i=0, row_count=171, ghost_row_count=0, pkey={tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337630621751, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.658876] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630658541, end_time=1727337630658876, runtime=335, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f927292dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C600-0-0, dag_ret:0, dag_status:2, start_time:1727337630657684, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.658872] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=101009, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b06060, count:4, allocator:0x7f92864987d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9265b06350, local_datums:0x7f9265b06350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:30.658886] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1202, dag_cnt=22, dag_cnts_[dag.get_type()]=22, &dag=0x7f927292dcb0, dag={ObIDag:{this:0x7f927292dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C600-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630657684, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.658894] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C600-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337630657682, task_id:YB42C0A8002D-0006230056D8C600-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.658909] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630658907, task_id:YB42C0A8002D-0006230056D8C605-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.658914] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=4] schedule one task(task=0x7f927292fa60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.658915] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.658939] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.658949] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.658952] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:30.658957] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.658950] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337630621769, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.658961] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.658961] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=41, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.658968] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.658968] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.658973] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.658972] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.658977] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.658988] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:377}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:43, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.658989] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337630621717, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.659015] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630658813, end_time=1727337630659015, runtime=202, *this={this:0x7f9272934ed0, type:1, status:2, dag:{ObIDag:{this:0x7f927292d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FE-0-0, dag_ret:0, dag_status:2, start_time:1727337630657672, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659007] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:377}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:377}, table_count:1, [{i:0, table_key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630657899, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.08ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.659026] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1354, dag_cnt=21, dag_cnts_[dag.get_type()]=21, &dag=0x7f927292d250, dag={ObIDag:{this:0x7f927292d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FE-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630657672, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.659029] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C601-0-0, dag_ret:0, dag_status:2, start_time:1727337630657689, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab480, output_block_cnt_arr:0x7f92684ab488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337630657899, estimated_finish_time:1727337647659029}) [2024-09-26 16:00:30.659035] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FE-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337630657668, task_id:YB42C0A8002D-0006230056D8C5FE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.659035] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f927292e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C601-0-0, dag_ret:0, dag_status:2, start_time:1727337630657689, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659040] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=5] task finish process(ret=0, start_time=1727337630657898, end_time=1727337630659040, runtime=1142, *this={this:0x7f927292e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f927292e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C601-0-0, dag_ret:0, dag_status:2, start_time:1727337630657689, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659071] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630659061, task_id:YB42C0A8002D-0006230056D8C606-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.659074] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=3] schedule one task(task=0x7f927292ff90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.659075] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337630621734, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.659091] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.659092] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337630621819, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.659100] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.659102] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.659102] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.659107] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101010}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:30.659122] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.659126] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.659120] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:101010}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101010}, table_count:1, [{i:0, table_key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8642d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630657963, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.14ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.659137] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FF-0-0, dag_ret:0, dag_status:2, start_time:1727337630657679, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db7470, output_block_cnt_arr:0x7f9287db7478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337630657963, estimated_finish_time:1727337647659137}) [2024-09-26 16:00:30.659131] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:333}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.659142] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f927292db40, type:15, status:2, dag:{ObIDag:{this:0x7f927292d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FF-0-0, dag_ret:0, dag_status:2, start_time:1727337630657679, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659147] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=5] task finish process(ret=0, start_time=1727337630657962, end_time=1727337630659147, runtime=1185, *this={this:0x7f927292db40, type:15, status:2, dag:{ObIDag:{this:0x7f927292d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FF-0-0, dag_ret:0, dag_status:2, start_time:1727337630657679, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659172] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.659175] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.659176] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:30.659180] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.659182] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.659188] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.659150] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:333}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:333}, table_count:1, [{i:0, table_key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8651da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630658689, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=440us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.659201] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337630621802, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.659209] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9263b5f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.659219] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.659216] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=53] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C604-0-0, dag_ret:0, dag_status:2, start_time:1727337630658677, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505490, output_block_cnt_arr:0x7f9266505498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337630658689, estimated_finish_time:1727337647659215}) [2024-09-26 16:00:30.659221] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.659223] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292f530, type:15, status:2, dag:{ObIDag:{this:0x7f927292f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C604-0-0, dag_ret:0, dag_status:2, start_time:1727337630658677, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659228] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=4] task finish process(ret=0, start_time=1727337630658687, end_time=1727337630659227, runtime=540, *this={this:0x7f927292f530, type:15, status:2, dag:{ObIDag:{this:0x7f927292f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C604-0-0, dag_ret:0, dag_status:2, start_time:1727337630658677, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659225] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263b5f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101065}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.659239] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:101065}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9263b5f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101065}, table_count:1, [{i:0, table_key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630658507, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=717us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.659255] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C603-0-0, dag_ret:0, dag_status:2, start_time:1727337630658450, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263b5f490, output_block_cnt_arr:0x7f9263b5f498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337630658507, estimated_finish_time:1727337647659255}) [2024-09-26 16:00:30.659260] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f927292f000, type:15, status:2, dag:{ObIDag:{this:0x7f927292ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C603-0-0, dag_ret:0, dag_status:2, start_time:1727337630658450, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659256] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337630621851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.659262] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=2] schedule one task(task=0x7f9262c043c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.659268] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=3] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.659265] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.659276] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.659264] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=4] task finish process(ret=0, start_time=1727337630658506, end_time=1727337630659264, runtime=758, *this={this:0x7f927292f000, type:15, status:2, dag:{ObIDag:{this:0x7f927292ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C603-0-0, dag_ret:0, dag_status:2, start_time:1727337630658450, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659278] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.659283] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101066}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.659299] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101066}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101066}, table_count:1, [{i:0, table_key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8653ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630658934, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=347us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.659313] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=2] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.659317] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=333, range=0x7f92a9ed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a64060, count:7, allocator:0x7f9275d767d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9264a643b0, local_datums:0x7f9264a643b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:30.659326] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C605-0-0, dag_ret:0, dag_status:2, start_time:1727337630658913, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad490, output_block_cnt_arr:0x7f92684ad498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337630658934, estimated_finish_time:1727337647659326}) [2024-09-26 16:00:30.659332] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292fa60, type:15, status:2, dag:{ObIDag:{this:0x7f927292f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C605-0-0, dag_ret:0, dag_status:2, start_time:1727337630658913, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659337] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=4] task finish process(ret=0, start_time=1727337630658932, end_time=1727337630659337, runtime=405, *this={this:0x7f927292fa60, type:15, status:2, dag:{ObIDag:{this:0x7f927292f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C605-0-0, dag_ret:0, dag_status:2, start_time:1727337630658913, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659367] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101065, range=0x7f92a9d52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5c060, count:6, allocator:0x7f9269ffa7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9263b5c390, local_datums:0x7f9263b5c390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:30.659439] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=41][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.659473] ERROR issue_dba_error (ob_log.cpp:1875) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.659476] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.659484] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=0] partition merge iter row count(i=0, row_count=15, ghost_row_count=0, pkey={tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337630621802, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.659496] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.659502] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630659332, end_time=1727337630659501, runtime=169, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f927292ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C603-0-0, dag_ret:0, dag_status:2, start_time:1727337630658450, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659510] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1060, dag_cnt=20, dag_cnts_[dag.get_type()]=20, &dag=0x7f927292ec40, dag={ObIDag:{this:0x7f927292ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C603-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630658450, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.659517] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C603-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630658445, task_id:YB42C0A8002D-0006230056D8C603-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.659533] ERROR issue_dba_error (ob_log.cpp:1875) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.659534] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=10] schedule one task(task=0x7f9272934ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.659539] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337630659538, task_id:YB42C0A8002D-0006230056D8C607-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.659536] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.659542] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=3] schedule one task(task=0x7f92729304c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.659546] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=0] partition merge iter row count(i=0, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337630621819, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.659582] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.659585] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.659580] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630659292, end_time=1727337630659580, runtime=288, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f927292f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C604-0-0, dag_ret:0, dag_status:2, start_time:1727337630658677, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659586] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:30.659590] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.659589] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=912, dag_cnt=19, dag_cnts_[dag.get_type()]=19, &dag=0x7f927292f170, dag={ObIDag:{this:0x7f927292f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C604-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630658677, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.659596] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C604-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630658672, task_id:YB42C0A8002D-0006230056D8C604-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.659603] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101066, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:7}, cols_desc:{ObIArray:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:7, allocator:0x7f92636707d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9263a383b0, local_datums:0x7f9263a383b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:30.659593] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077691154184001, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8650e00, is_inited:true, wash_score:1727337630621848000, hold_ref_cnt:true, ref_cnt:2, version:2, length:497, tablet_addr:[809012-1445-0-2305843637353218048-2](offset=557056,size=585,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:451}, data_tablet_id:{id:451}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077691154184001, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2119849913, row_count:31}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86514b8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844404004851712-0](offset=528384,size=1299,type=2,seq=0), type:"MAJOR", tablet_id:{id:451}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563069147000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843980950574355-0](offset=529683,size=905,type=2,seq=0), type:"MINI", tablet_id:{id:451}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986382094000, upper_trans_version:1726149986382094000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337630621892, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdfc00, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334222657108}, this:0x7f92a3bdfc00, timestamp:1727334222657108, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753331112, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:89, clock:641728512}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293246986000, v:0}, rec_scn:{val:1726265293246986000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696516096-2](offset=536576,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843170275504460-0](offset=536908,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879883234-0](offset=537058,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446403268-0](offset=537284,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc4080, 0x7f92a3bdfc00, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.659631] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=37] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.659636] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.659666] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101010, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d60060, count:4, allocator:0x7f92bc90a7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264d60350, local_datums:0x7f9264d60350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:30.659700] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.659731] ERROR issue_dba_error (ob_log.cpp:1875) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.659733] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.659743] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=21][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.659741] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337630621734, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.659760] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630659622, end_time=1727337630659760, runtime=138, *this={this:0x7f9262c043c0, type:1, status:2, dag:{ObIDag:{this:0x7f927292d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FF-0-0, dag_ret:0, dag_status:2, start_time:1727337630657679, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659768] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2089, dag_cnt=18, dag_cnts_[dag.get_type()]=18, &dag=0x7f927292d780, dag={ObIDag:{this:0x7f927292d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C5FF-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630657679, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.659775] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.659776] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C5FF-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630657676, task_id:YB42C0A8002D-0006230056D8C5FF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.659778] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.659786] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=0] partition merge iter row count(i=0, row_count=182, ghost_row_count=0, pkey={tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337630621851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.659804] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630659577, end_time=1727337630659804, runtime=227, *this={this:0x7f9272934ed0, type:1, status:2, dag:{ObIDag:{this:0x7f927292f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C605-0-0, dag_ret:0, dag_status:2, start_time:1727337630658913, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659812] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=899, dag_cnt=17, dag_cnts_[dag.get_type()]=17, &dag=0x7f927292f6a0, dag={ObIDag:{this:0x7f927292f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C605-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630658913, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.659819] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C605-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630658907, task_id:YB42C0A8002D-0006230056D8C605-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.659817] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337630621871, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.659825] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=19, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.659845] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337630659844, task_id:YB42C0A8002D-0006230056D8C608-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.659848] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=3] schedule one task(task=0x7f92729309f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.659852] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630659851, task_id:YB42C0A8002D-0006230056D8C609-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.659854] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=2] schedule one task(task=0x7f9272930f20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.659859] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337630659858, task_id:YB42C0A8002D-0006230056D8C60A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.659861] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=2] schedule one task(task=0x7f9272931450, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.659864] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.659868] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.659881] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.659884] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.659886] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:30.659872] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:334}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:21, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.659888] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.659889] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.659892] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.659892] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.659894] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:30.659897] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241548806527000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.659898] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.659892] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:334}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:334}, table_count:1, [{i:0, table_key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8654c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630659167, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=702us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.659901] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.659916] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=11] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.659966] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337630621892, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.659974] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.659976] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.659971] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C606-0-0, dag_ret:0, dag_status:2, start_time:1727337630659074, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684b14a0, output_block_cnt_arr:0x7f92684b14a8, concurrent_cnt:1, estimate_row_cnt:993, estimate_occupy_size:0, latest_update_ts:1727337630659167, estimated_finish_time:1727337647659970}) [2024-09-26 16:00:30.659978] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:30.659975] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f92664ef330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=14, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.659981] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.659979] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f927292ff90, type:15, status:2, dag:{ObIDag:{this:0x7f927292fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C606-0-0, dag_ret:0, dag_status:2, start_time:1727337630659074, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659984] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.659985] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.659984] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=5] task finish process(ret=0, start_time=1727337630659166, end_time=1727337630659984, runtime=818, *this={this:0x7f927292ff90, type:15, status:2, dag:{ObIDag:{this:0x7f927292fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C606-0-0, dag_ret:0, dag_status:2, start_time:1727337630659074, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.659989] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.659988] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.659993] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92664ef330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:451}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:16, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.660008] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:451}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f92664ef330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:451}, table_count:1, [{i:0, table_key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8650e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630659578, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=413us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.660027] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=2] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.660031] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=3] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.660028] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272930100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C607-0-0, dag_ret:0, dag_status:2, start_time:1727337630659542, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92664ef470, output_block_cnt_arr:0x7f92664ef478, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337630659578, estimated_finish_time:1727337647660028}) [2024-09-26 16:00:30.660034] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f92729304c0, type:15, status:2, dag:{ObIDag:{this:0x7f9272930100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C607-0-0, dag_ret:0, dag_status:2, start_time:1727337630659542, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.660039] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=4] task finish process(ret=0, start_time=1727337630659576, end_time=1727337630660039, runtime=463, *this={this:0x7f92729304c0, type:15, status:2, dag:{ObIDag:{this:0x7f9272930100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C607-0-0, dag_ret:0, dag_status:2, start_time:1727337630659542, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.660064] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=377, range=0x7f92aad52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:41, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:43}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:5, allocator:0x7f92b323e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:43, datum_buffer:{capacity:43, datums:0x7f9263b5b2b0, local_datums:0x7f9263b5a360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,]}) [2024-09-26 16:00:30.660081] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=334, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:19, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:21}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c76060, count:8, allocator:0x7f928649a7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:21, datum_buffer:{capacity:32, datums:0x7f9262c763e0, local_datums:0x7f9262c763e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,]}) [2024-09-26 16:00:30.660101] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.660132] ERROR issue_dba_error (ob_log.cpp:1875) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.660135] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.660158] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=0] partition merge iter row count(i=0, row_count=6, ghost_row_count=0, pkey={tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337630621769, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.660176] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630660038, end_time=1727337630660176, runtime=138, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f927292e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C601-0-0, dag_ret:0, dag_status:2, start_time:1727337630657689, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.660184] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2495, dag_cnt=16, dag_cnts_[dag.get_type()]=16, &dag=0x7f927292e1e0, dag={ObIDag:{this:0x7f927292e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C601-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630657689, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.660191] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C601-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630657687, task_id:YB42C0A8002D-0006230056D8C601-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.660364] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.660400] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.660403] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.660413] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=0] partition merge iter row count(i=0, row_count=153, ghost_row_count=0, pkey={tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337630621871, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.660448] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630660040, end_time=1727337630660448, runtime=408, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f927292fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C606-0-0, dag_ret:0, dag_status:2, start_time:1727337630659074, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.660457] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1383, dag_cnt=15, dag_cnts_[dag.get_type()]=15, &dag=0x7f927292fbd0, dag={ObIDag:{this:0x7f927292fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C606-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630659074, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.660465] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C606-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337630659061, task_id:YB42C0A8002D-0006230056D8C606-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.660493] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752511928, storage_schema:{this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.660502] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.660505] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.660503] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337630621943, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.660512] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752725472, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752725472, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.660510] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:121}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752511928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.660523] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.660526] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.660524] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:121}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630659877, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=631us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.660531] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:215}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752725472, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.660541] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C609-0-0, dag_ret:0, dag_status:2, start_time:1727337630659854, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f92685eb480, output_block_cnt_arr:0x7f92685eb488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337630659877, estimated_finish_time:1727337647660541}) [2024-09-26 16:00:30.660546] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272930f20, type:15, status:2, dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C609-0-0, dag_ret:0, dag_status:2, start_time:1727337630659854, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:30.660551] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=5] task finish process(ret=0, start_time=1727337630659876, end_time=1727337630660551, runtime=675, *this={this:0x7f9272930f20, type:15, status:2, dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C609-0-0, dag_ret:0, dag_status:2, start_time:1727337630659854, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:30.660546] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:215}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752725472, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:215}, table_count:1, [{i:0, table_key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630659883, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=646us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.660566] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272931090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60A-0-0, dag_ret:0, dag_status:2, start_time:1727337630659861, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505460, output_block_cnt_arr:0x7f9266505468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337630659883, estimated_finish_time:1727337647660565}) [2024-09-26 16:00:30.660571] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272931450, type:15, status:2, dag:{ObIDag:{this:0x7f9272931090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60A-0-0, dag_ret:0, dag_status:2, start_time:1727337630659861, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.660576] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=5] task finish process(ret=0, start_time=1727337630659881, end_time=1727337630660576, runtime=695, *this={this:0x7f9272931450, type:15, status:2, dag:{ObIDag:{this:0x7f9272931090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60A-0-0, dag_ret:0, dag_status:2, start_time:1727337630659861, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.660590] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337630621909, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.660620] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=30] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.660629] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.660632] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.660636] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:378}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.660650] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:378}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:378}, table_count:1, [{i:0, table_key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630659970, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=664us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.660666] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272930630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C608-0-0, dag_ret:0, dag_status:2, start_time:1727337630659848, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad480, output_block_cnt_arr:0x7f92684ad488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337630659970, estimated_finish_time:1727337647660666}) [2024-09-26 16:00:30.660671] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f92729309f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272930630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C608-0-0, dag_ret:0, dag_status:2, start_time:1727337630659848, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.660675] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=4] task finish process(ret=0, start_time=1727337630659969, end_time=1727337630660675, runtime=706, *this={this:0x7f92729309f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272930630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C608-0-0, dag_ret:0, dag_status:2, start_time:1727337630659848, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.660690] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=2] schedule one task(task=0x7f9262cbe030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.660694] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=3] schedule one task(task=0x7f9262cde030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.660697] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=2] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.660699] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=1] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.660705] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630660704, task_id:YB42C0A8002D-0006230056D8C60B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.660709] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=3] schedule one task(task=0x7f9272931980, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.660712] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337630660712, task_id:YB42C0A8002D-0006230056D8C60C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.660714] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=2] schedule one task(task=0x7f9272931eb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.660730] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.660734] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.660736] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:30.660740] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.660742] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.660733] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=378, range=0x7f92aa952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92728f0060, count:5, allocator:0x7f9269ffa7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92728f0360, local_datums:0x7f92728f0360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:30.660748] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.660778] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=451, range=0x7f92a9cd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:14, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:16}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665ae060, count:4, allocator:0x7f92bc9127d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:16, datum_buffer:{capacity:32, datums:0x7f92665ae350, local_datums:0x7f92665ae350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,]}) [2024-09-26 16:00:30.660794] ERROR issue_dba_error (ob_log.cpp:1875) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.660797] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.660804] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.660796] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=121, range=0x7f92aaad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b58060, count:5, allocator:0x7f92a33567d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263b58360, local_datums:0x7f9263b58360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:30.660807] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337630621909, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.660813] INFO [STORAGE] dump_info (ob_sstable_struct.cpp:262) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=14] dump merge info(msg="macro block builder close", output_row_per_s=0, new_macro_KB_per_s=0, *this={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:40, merge_start_time:1727337630660773, merge_finish_time:1727337630660813, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:0, max:0, sum:0, count:0}}, {type:"cost_time", info:{min:0, max:0, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:0, max:0, sum:0, count:0}}, {type:"incremental_row_count", info:{min:0, max:0, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:0, snapshot_version:0, start_scn:0, end_scn:0}, macro_id_list:"", comment:""}) [2024-09-26 16:00:30.660825] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630660703, end_time=1727337630660825, runtime=122, *this={this:0x7f9262cde030, type:1, status:2, dag:{ObIDag:{this:0x7f9272930630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C608-0-0, dag_ret:0, dag_status:2, start_time:1727337630659848, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.660828] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=14] partition merge iter row count(i=0, row_count=0, ghost_row_count=0, pkey={tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, table={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337630621926, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.660836] ERROR issue_dba_error (ob_log.cpp:1875) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.660834] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=986, dag_cnt=14, dag_cnts_[dag.get_type()]=14, &dag=0x7f9272930630, dag={ObIDag:{this:0x7f9272930630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C608-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630659848, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.660838] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.660839] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1544) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=9] merge macro blocks ok(idx_=0, task={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C609-0-0, dag_ret:0, dag_status:2, start_time:1727337630659854, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:30.660842] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C608-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337630659844, task_id:YB42C0A8002D-0006230056D8C608-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.660849] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=6] task finish process(ret=0, start_time=1727337630660770, end_time=1727337630660849, runtime=79, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C609-0-0, dag_ret:0, dag_status:2, start_time:1727337630659854, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:30.660847] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337630621892, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.660876] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630660702, end_time=1727337630660876, runtime=174, *this={this:0x7f9262cbe030, type:1, status:2, dag:{ObIDag:{this:0x7f9272930100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C607-0-0, dag_ret:0, dag_status:2, start_time:1727337630659542, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.660882] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=2] schedule one task(task=0x7f9262c04250, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.660888] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337630660887, task_id:YB42C0A8002D-0006230056D8C60D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.660891] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=3] schedule one task(task=0x7f92729323e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.660912] INFO [STORAGE.COMPACTION] get_merged_sstable (ob_tablet_merge_task.cpp:1285) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=2] create new merged sstable(ctx.param_.tablet_id_={id:121}, snapshot_version=1726241132764173999, ctx.param_.merge_type_=3, ctx.create_snapshot_version_=0, table_mode_flag=0) [2024-09-26 16:00:30.660917] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.660920] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.660922] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:30.660926] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.660919] INFO [STORAGE] close (ob_index_block_builder.cpp:815) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=3] succeed to close sstable index builder(res={root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_block_ids_.count():0, other_block_ids_.count():0, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, data_column_cnt:10, data_column_checksums:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], row_count:0, max_merged_trans_version:0, contain_uncommitted_row:false, occupy_size:0, original_size:0, data_checksum:0, use_old_macro_block_count:0, compressor_type:1, root_row_store_type:0, nested_offset:0, nested_size:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}) [2024-09-26 16:00:30.660936] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=8][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.660940] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.660943] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:30.660948] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.660935] INFO [STORAGE] init (ob_sstable.cpp:118) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=9] succeeded to init sstable(ret=0, this={ObITable:{this:0x7f92685ea128, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f92685ea128, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:null, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:30.660929] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b863a0a0, is_inited:true, wash_score:1727337630621848000, hold_ref_cnt:true, ref_cnt:2, version:2, length:499, tablet_addr:[816017-1849-0-2305843639502073856-2](offset=1929216,size=587,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101015}, data_tablet_id:{id:154}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726074677203981000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:693685402, row_count:578}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863a7a0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844390047592448-0](offset=1912832,size=1286,type=2,seq=0), type:"MAJOR", tablet_id:{id:101015}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726074015354550000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843987394409734-0](offset=1914118,size=911,type=2,seq=0), type:"MINOR", tablet_id:{id:101015}, scn_range:{start_scn:{val:1726074677203981000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726205802467242000, upper_trans_version:1726205802467242000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337630622005, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd57f0, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209508621}, this:0x7f92a3bd57f0, timestamp:1727334209508621, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:68, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277803107894000, v:0}, rec_scn:{val:1726241804194470000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771638272-2](offset=1916928,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161786701-0](offset=1917261,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881263599-0](offset=1917423,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc7730, 0x7f92a3bd57f0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.660966] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=37] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.660953] INFO [STORAGE.COMPACTION] create_sstable (ob_tablet_merge_ctx.cpp:367) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=18] succeed to merge sstable(param={table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_logic_seq:0, schema_version:1710828752511928, create_snapshot_version:0, progressive_merge_round:0, progressive_merge_step:0, is_ready_for_read:true, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, index_type:0, root_block_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), root_block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, root_row_store_type:0, latest_row_store_type:1, data_index_tree_height:0, data_block_macro_meta_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), data_block_macro_meta:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, use_old_macro_block_count:0, row_count:0, rowkey_column_cnt:5, column_cnt:10, column_checksums:[], data_checksum:0, occupy_size:0, original_size:0, max_merged_trans_version:1726241132764173999, ddl_scn:{val:0, v:0}, contain_uncommitted_row:false, is_meta_root:false, compressor_type:1, encrypt_id:0, master_key_id:0, recycle_version:0, nested_offset:0, nested_size:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}, table_key={tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_merge_info={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:936, merge_start_time:1727337630659877, merge_finish_time:1727337630660813, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"cost_time", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"incremental_row_count", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:1, snapshot_version:0, start_scn:1726211867305630000, end_scn:1726241548806527000}, macro_id_list:"", comment:""}) [2024-09-26 16:00:30.660972] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.660952] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8623ce0, is_inited:true, wash_score:1727337630621848000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[816017-1849-0-2305843644870885376-2](offset=2031616,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:154}, data_tablet_id:{id:154}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726074677203981000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3786676695, row_count:578}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86243bc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844492053168128-0](offset=2015232,size=1381,type=2,seq=0), type:"MAJOR", tablet_id:{id:154}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726074015354550000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843989541995877-0](offset=2016613,size=913,type=2,seq=0), type:"MINOR", tablet_id:{id:154}, scn_range:{start_scn:{val:1726074677203981000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726205802467242000, upper_trans_version:1726205802467242000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337630621987, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd5020, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209508489}, this:0x7f92a3bd5020, timestamp:1727334209508489, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:67, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277803107894000, v:0}, rec_scn:{val:1726241804194470000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843367845486592-2](offset=2023424,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719442254-0](offset=2023758,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881370090-0](offset=2023914,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447890124-0](offset=2024140,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc6f60, 0x7f92a3bd5020, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.660999] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=47] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.661006] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.661101] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=215, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:6}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:3, allocator:0x7f92636107d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9263a38320, local_datums:0x7f9263a38320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:30.661170] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337630622005, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.661176] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.661179] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.661179] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.661189] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.661192] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.661188] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=0] partition merge iter row count(i=0, row_count=9, ghost_row_count=0, pkey={tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337630621943, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.661197] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101015}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.661206] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630661073, end_time=1727337630661206, runtime=133, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9272931090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60A-0-0, dag_ret:0, dag_status:2, start_time:1727337630659861, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.661219] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1358, dag_cnt=13, dag_cnts_[dag.get_type()]=13, &dag=0x7f9272931090, dag={ObIDag:{this:0x7f9272931090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630659861, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.661227] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60A-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337630659858, task_id:YB42C0A8002D-0006230056D8C60A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.661216] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:101015}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101015}, table_count:1, [{i:0, table_key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630660912, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=284us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.661237] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272932020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60D-0-0, dag_ret:0, dag_status:2, start_time:1727337630660891, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f490, output_block_cnt_arr:0x7f9264b1f498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337630660912, estimated_finish_time:1727337647661237}) [2024-09-26 16:00:30.661243] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f92729323e0, type:15, status:2, dag:{ObIDag:{this:0x7f9272932020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60D-0-0, dag_ret:0, dag_status:2, start_time:1727337630660891, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.661247] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=4] task finish process(ret=0, start_time=1727337630660910, end_time=1727337630661247, runtime=337, *this={this:0x7f92729323e0, type:15, status:2, dag:{ObIDag:{this:0x7f9272932020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60D-0-0, dag_ret:0, dag_status:2, start_time:1727337630660891, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.661258] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1716, dag_cnt=12, dag_cnts_[dag.get_type()]=12, &dag=0x7f9272930100, dag={ObIDag:{this:0x7f9272930100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C607-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630659542, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.661265] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C607-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337630659538, task_id:YB42C0A8002D-0006230056D8C607-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.661300] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337630621987, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.661311] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.661341] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=26] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.661344] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.661348] INFO [STORAGE] build_new_table_store (ob_tablet_table_store.cpp:1206) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=18] succeed to build new table store(major_tables_={cnt:1, is_inited:true}, minor_tables_={cnt:3, is_inited:true}, memtables_={count:1}, PRINT_TS(*this)={ObTabletTableStore_Pretty:this:0x7f92ab151ca8, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, memtables:{count:1}, is_ready_for_read:true, table_array:{ [2850] [ YB42C0A8002D-0006230056D8C609-0-0 ] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [2850] [ YB42C0A8002D-0006230056D8C609-0-0 ] MAJOR MAJOR 1725120003420849000 1725120003420849000 {val:0, v:0} {val:1726077600415496000, v:0} 0 false [2850] [ YB42C0A8002D-0006230056D8C609-0-0 ] MINOR MINOR 1726178585269925002 1726178585269925002 {val:1726077259159067000, v:0} {val:1726178585269925002, v:0} 0 false [2850] [ YB42C0A8002D-0006230056D8C609-0-0 ] MINI 1726211867106157000 1726211867106157000 {val:1726178585269925002, v:0} {val:1726211867305630000, v:0} 0 false [2850] [ YB42C0A8002D-0006230056D8C609-0-0 ] MINI 1726241132764173999 1726241132764173999 {val:1726211867305630000, v:0} {val:1726241548806527000, v:0} 0 false [2850] [ YB42C0A8002D-0006230056D8C609-0-0 ] MEM ACTIVE 9223372036854775807 18446744073709547609 {val:1726241548806527000, v:0} {val:4611686018427387903, v:0} 3 unused }}) [2024-09-26 16:00:30.661350] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:154}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.661367] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:154}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:154}, table_count:1, [{i:0, table_key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8623ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630660929, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=419us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.661390] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272931af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60C-0-0, dag_ret:0, dag_status:2, start_time:1727337630660714, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad480, output_block_cnt_arr:0x7f92684ad488, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337630660929, estimated_finish_time:1727337647661390}) [2024-09-26 16:00:30.661396] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272931eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9272931af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60C-0-0, dag_ret:0, dag_status:2, start_time:1727337630660714, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.661400] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=4] task finish process(ret=0, start_time=1727337630660927, end_time=1727337630661400, runtime=473, *this={this:0x7f9272931eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9272931af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60C-0-0, dag_ret:0, dag_status:2, start_time:1727337630660714, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.661409] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=2] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.661416] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=4] schedule one task(task=0x7f9262c043c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.661420] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630661419, task_id:YB42C0A8002D-0006230056D8C60E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.661424] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=3] schedule one task(task=0x7f9272932910, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.661440] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] succeed to add sys task(task={start_time:1727337630661439, task_id:YB42C0A8002D-0006230056D8C60F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.661442] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=2] schedule one task(task=0x7f9272932e40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.661469] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.661472] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.661474] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:30.661478] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.661481] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.661487] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.661543] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.661547] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.661536] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=154, range=0x7f92aaed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a60060, count:5, allocator:0x7f92bc9127d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a60360, local_datums:0x7f9264a60360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:30.661548] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:30.661552] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.661554] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.661559] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.661600] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.661632] ERROR issue_dba_error (ob_log.cpp:1875) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.661635] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.661631] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101015, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fa060, count:6, allocator:0x7f92691327d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92665fa390, local_datums:0x7f92665fa390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:30.661642] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337630621987, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.661659] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630661480, end_time=1727337630661659, runtime=179, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9272931af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60C-0-0, dag_ret:0, dag_status:2, start_time:1727337630660714, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.661668] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=954, dag_cnt=11, dag_cnts_[dag.get_type()]=11, &dag=0x7f9272931af0, dag={ObIDag:{this:0x7f9272931af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630660714, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.661674] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60C-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630660712, task_id:YB42C0A8002D-0006230056D8C60C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.661675] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.661763] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.661766] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.661776] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337630622005, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.661796] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630661602, end_time=1727337630661796, runtime=194, *this={this:0x7f9262c043c0, type:1, status:2, dag:{ObIDag:{this:0x7f9272932020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60D-0-0, dag_ret:0, dag_status:2, start_time:1727337630660891, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.661835] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=944, dag_cnt=10, dag_cnts_[dag.get_type()]=10, &dag=0x7f9272932020, dag={ObIDag:{this:0x7f9272932020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630660891, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.661843] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60D-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630660887, task_id:YB42C0A8002D-0006230056D8C60D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.661889] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337630622039, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.661899] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.661943] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=33] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.661947] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.661956] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101063}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.661981] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=19] succeed to build merge ctx(tablet_id={id:101063}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101063}, table_count:1, [{i:0, table_key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630661540, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=412us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.662006] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272932a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60F-0-0, dag_ret:0, dag_status:2, start_time:1727337630661442, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f14a0, output_block_cnt_arr:0x7f92728f14a8, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337630661540, estimated_finish_time:1727337647662006}) [2024-09-26 16:00:30.662012] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272932e40, type:15, status:2, dag:{ObIDag:{this:0x7f9272932a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60F-0-0, dag_ret:0, dag_status:2, start_time:1727337630661442, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.662017] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=5] task finish process(ret=0, start_time=1727337630661538, end_time=1727337630662017, runtime=479, *this={this:0x7f9272932e40, type:15, status:2, dag:{ObIDag:{this:0x7f9272932a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60F-0-0, dag_ret:0, dag_status:2, start_time:1727337630661442, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.662055] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337630622022, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.662064] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.662074] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.662076] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=1] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.662080] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101016}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.662095] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101016}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101016}, table_count:1, [{i:0, table_key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630661465, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=613us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.662114] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272932550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60E-0-0, dag_ret:0, dag_status:2, start_time:1727337630661423, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5490, output_block_cnt_arr:0x7f9287db5498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337630661465, estimated_finish_time:1727337647662113}) [2024-09-26 16:00:30.662119] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272932910, type:15, status:2, dag:{ObIDag:{this:0x7f9272932550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60E-0-0, dag_ret:0, dag_status:2, start_time:1727337630661423, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.662123] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=4] task finish process(ret=0, start_time=1727337630661463, end_time=1727337630662123, runtime=660, *this={this:0x7f9272932910, type:15, status:2, dag:{ObIDag:{this:0x7f9272932550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60E-0-0, dag_ret:0, dag_status:2, start_time:1727337630661423, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.662138] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337630621960, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.662146] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753355744, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753355744, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.662154] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.662156] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.662160] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:456}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753355744, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.662173] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:456}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753355744, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:456}, table_count:1, [{i:0, table_key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8634c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630660726, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.43ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.662191] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92729315c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60B-0-0, dag_ret:0, dag_status:2, start_time:1727337630660709, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d490, output_block_cnt_arr:0x7f9264b1d498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337630660726, estimated_finish_time:1727337647662191}) [2024-09-26 16:00:30.662196] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272931980, type:15, status:2, dag:{ObIDag:{this:0x7f92729315c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60B-0-0, dag_ret:0, dag_status:2, start_time:1727337630660709, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.662200] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=4] task finish process(ret=0, start_time=1727337630660724, end_time=1727337630662200, runtime=1476, *this={this:0x7f9272931980, type:15, status:2, dag:{ObIDag:{this:0x7f92729315c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60B-0-0, dag_ret:0, dag_status:2, start_time:1727337630660709, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.662214] INFO [MDS] init_for_evict_medium_info (ob_tablet_mds_data.cpp:414) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=15] succeeded to init mds data(ret=0, finish_medium_scn=1726077600415496000, src_medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, src_extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}) [2024-09-26 16:00:30.662230] INFO [STORAGE] init_for_merge (ob_tablet.cpp:363) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=6] succeeded to init tablet for mini/minor/major merge(ret=0, param={sstable:0x7f92685ea128, snapshot_version:1726241132764173999, clog_checkpoint_scn:{val:1726241548806527000, v:0}, multi_version_start:1726178585269925002, need_report:false, storage_schema:{this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, rebuild_seq:0, update_with_major_flag:false, need_check_sstable:false, ddl_info:{keep_old_ddl_sstable:true, ddl_start_scn:{val:0, v:0}, ddl_commit_scn:{val:0, v:0}, ddl_checkpoint_scn:{val:0, v:0}, ddl_snapshot_version:0, ddl_execution_id:-1, data_format_version:0}, allow_duplicate_sstable:false, merge_type:"MINI_MERGE", need_check_transfer_seq:true, transfer_seq:0}, old_tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337630621848000, hold_ref_cnt:true, ref_cnt:3, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337630621926, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}, this={this:0x7f9264b20060, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x7f92ab152450, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726241548806527000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726241132764173999, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f9264b20670, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}, {i:2, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726241132764173999, upper_trans_version:1726241132764173999}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x7f9264b20520, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9264b21880, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9264b21950, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{redefined:false, snapshot_version:0, schema_version:0, data_tablet_id:{id:0}, hidden_tablet_id:{id:0}, lob_meta_tablet_id:{id:50121}, lob_piece_tablet_id:{id:60121}, is_old_mds:true}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:30.662320] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=76] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287db6060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}}, ref_cnt:0, upper_trans_version:1725120003420849000, timestamp:0}, this:0x7f9287db6060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1725120003420849000, max_merged_trans_version:1725120003420849000, data_macro_block_count:1, nested_size:4096, nested_offset:589824, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:84, row_count_:14, occupy_size_:1604, original_size_:1914, data_checksum_:3587602083, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:1, data_micro_block_count_:1, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:2, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1725120003420849000, max_merged_trans_version_:1725120003420849000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287db6310, column_checksum_count:10, data_root_info:{addr:[0-0-0-3458764933653594112-0](offset=0,size=391,type=3,seq=0), block_data:{buf:0x7f9287db6360, size:391, extra_buf:0x7f9287db64e7, extra_size:424, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-3458765009889263616-0](offset=0,size=462,type=3,seq=0), block_data:{buf:0x7f9287db668f, size:462, extra_buf:null, extra_size:0, type:1}}, data_block_count:1, other_block_count:0, linked_block_count:0, data_block_ids:0x7f9287db685d, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:true, nested_offset:589824, nested_size:4096}}, valid_for_reading:true}) [2024-09-26 16:00:30.662339] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=17] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287db6060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}}, ref_cnt:0, upper_trans_version:1726178585269925002, timestamp:0}, this:0x7f9287db6060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726178585269925002, max_merged_trans_version:1726178585269925002, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726178585269925002, v:0}, meta:{basic_meta:{version:1, length:94, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726178585269925002, max_merged_trans_version_:1726178585269925002, recycle_version:1726077600415496000, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726178585269925002, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:1, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287db6310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:30.662350] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=10] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287db6060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}}, ref_cnt:0, upper_trans_version:1726211867106157000, timestamp:0}, this:0x7f9287db6060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726211867106157000, max_merged_trans_version:1726211867106157000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726211867305630000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726211867106157000, max_merged_trans_version_:1726211867106157000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726211867305630000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287db6310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:30.662360] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=9] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287db6060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9287db6060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287db6310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:30.662401] ERROR issue_dba_error (ob_log.cpp:1875) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.662403] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf9de5b1 0xf9dcfd6 0xe51c6fb 0xe519c03 0xe518f89 0xe516f8c 0xf01158d 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.662420] INFO [STORAGE] reset (ob_tablet.cpp:159) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=0] reset tablet(this=0x7f9264b20060, ls_id={id:1}, tablet_id={id:121}, lbt()="0x121f1f9c 0xe52cc09 0xe52c6c5 0xe52bd13 0x4fd348f 0xf01110c 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:30.662426] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1250) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=0][errcode=-4184] sstable merge finish(ret=-4184, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:true, merge_dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C609-0-0, dag_ret:0, dag_status:2, start_time:1727337630659854, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f92685eb480, output_block_cnt_arr:0x7f92685eb488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337630659877, estimated_finish_time:1727337647660541}, compaction_filter:NULL, time_guard:total=1.09ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, task={this:0x7f9262c04250, type:16, status:2, dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C609-0-0, dag_ret:0, dag_status:2, start_time:1727337630659854, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:30.662448] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630660910, end_time=1727337630662447, runtime=1537, *this={this:0x7f9262c04250, type:16, status:2, dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C609-0-0, dag_ret:0, dag_status:2, start_time:1727337630659854, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:30.662454] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2600, dag_cnt=9, dag_cnts_[dag.get_type()]=9, &dag=0x7f9272930b60, dag={ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C609-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630659854, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}) [2024-09-26 16:00:30.662461] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C609-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630659851, task_id:YB42C0A8002D-0006230056D8C609-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.662511] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=2] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.662516] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=4] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.662521] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=2] schedule one task(task=0x7f9264d76030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.662525] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630662524, task_id:YB42C0A8002D-0006230056D8C610-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.662528] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=3] schedule one task(task=0x7f9272933370, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.662531] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630662531, task_id:YB42C0A8002D-0006230056D8C611-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.662534] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=2] schedule one task(task=0x7f92729338a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.662537] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337630662536, task_id:YB42C0A8002D-0006230056D8C612-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.662538] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=1] schedule one task(task=0x7f9272933dd0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.662554] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.662557] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.662559] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:30.662563] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.662566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.662573] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.662591] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.662593] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.662594] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:30.662597] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.662594] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101063, range=0x7f92a9b52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d30060, count:8, allocator:0x7f9269ffc7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d303e0, local_datums:0x7f9287d303e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:30.662599] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706139416032, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f926e072bc0, is_inited:true, wash_score:1727337630621848000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[812168-2208-0-2305843644869693440-2](offset=839680,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:328}, data_tablet_id:{id:328}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706139416032, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:867269498, row_count:786}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f926e07329c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[812168-2208-0-2305844589762490368-0](offset=831488,size=1472,type=2,seq=0), type:"MAJOR", tablet_id:{id:328}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562828712000}}, minor_tables:{{i:0, addr:[812168-2208-0-2305843935853721024-0](offset=832960,size=863,type=2,seq=0), type:"MINI", tablet_id:{id:328}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986144852000, upper_trans_version:1726149986144852000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337630622073, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bda610, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220621826}, this:0x7f92a3bda610, timestamp:1727334220621826, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:78, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265257450254001, v:0}, rec_scn:{val:1726264907253071001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[812168-2208-0-2305843366770556928-2](offset=835584,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[812168-2208-0-2305843176718254413-0](offset=835917,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843251880182249-0](offset=836073,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843216446702283-0](offset=836299,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc8ea0, 0x7f92a3bda610, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.662634] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=35] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.662638] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.662651] ERROR issue_dba_error (ob_log.cpp:1875) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.662654] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.662662] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337630622039, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.662680] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630662564, end_time=1727337630662680, runtime=116, *this={this:0x7f9264d76030, type:1, status:2, dag:{ObIDag:{this:0x7f9272932a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60F-0-0, dag_ret:0, dag_status:2, start_time:1727337630661442, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.662677] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=456, range=0x7f92ab0d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a36060, count:7, allocator:0x7f9275d787d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f9263a363b0, local_datums:0x7f9263a363b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:30.662688] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1246, dag_cnt=8, dag_cnts_[dag.get_type()]=8, &dag=0x7f9272932a80, dag={ObIDag:{this:0x7f9272932a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630661442, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.662696] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60F-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337630661439, task_id:YB42C0A8002D-0006230056D8C60F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.662755] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.662753] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101016, range=0x7f92aae56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92729f2060, count:6, allocator:0x7f92636467d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92729f2390, local_datums:0x7f92729f2390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:30.662800] ERROR issue_dba_error (ob_log.cpp:1875) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.662803] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.662812] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=0] partition merge iter row count(i=0, row_count=45, ghost_row_count=0, pkey={tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337630621960, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.662830] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630662651, end_time=1727337630662830, runtime=179, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f92729315c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60B-0-0, dag_ret:0, dag_status:2, start_time:1727337630660709, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.662839] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2130, dag_cnt=7, dag_cnts_[dag.get_type()]=7, &dag=0x7f92729315c0, dag={ObIDag:{this:0x7f92729315c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630660709, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.662845] ERROR issue_dba_error (ob_log.cpp:1875) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.662846] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630660704, task_id:YB42C0A8002D-0006230056D8C60B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.662847] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.662866] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337630622022, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.662884] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337630662721, end_time=1727337630662884, runtime=163, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9272932550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60E-0-0, dag_ret:0, dag_status:2, start_time:1727337630661423, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.662892] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1469, dag_cnt=6, dag_cnts_[dag.get_type()]=6, &dag=0x7f9272932550, dag={ObIDag:{this:0x7f9272932550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C60E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630661423, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.662898] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C60E-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630661419, task_id:YB42C0A8002D-0006230056D8C60E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.662916] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.662919] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.662921] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:30.662925] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.662927] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8652d40, is_inited:true, wash_score:1727337630621848000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[816017-1849-0-2305843644870942720-2](offset=2088960,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:452}, data_tablet_id:{id:452}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2139885871, row_count:290}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b865341c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844508159356928-0](offset=2076672,size=1396,type=2,seq=0), type:"MAJOR", tablet_id:{id:452}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562830808000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843992763282804-0](offset=2078068,size=916,type=2,seq=0), type:"MINI", tablet_id:{id:452}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986149768000, upper_trans_version:1726149986149768000}, {i:1, addr:[816017-1849-0-2305844002426960136-0](offset=2078984,size=925,type=2,seq=0), type:"MINI", tablet_id:{id:452}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726208018712107000, upper_trans_version:1726208018712107000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337630622055, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd9670, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220578716}, this:0x7f92a3bd9670, timestamp:1727334220578716, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:76, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265292959510000, v:0}, rec_scn:{val:1726264902486897000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843480588435456-2](offset=2080768,size=439,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719499703-0](offset=2081207,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881427539-0](offset=2081363,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447947573-0](offset=2081589,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc9e40, 0x7f92a3bd9670, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:30.662963] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=36] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.662968] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.662984] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337630622073, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.662992] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=7] update column cnt(column_cnt_=26, store_column_cnt_=26, column_cnt_=26, column_array_.count()=0) [2024-09-26 16:00:30.662994] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9287d2f480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=26, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.663005] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.663008] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.663013] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287d2f480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:328}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:28, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.663029] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:328}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9287d2f480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:328}, table_count:1, [{i:0, table_key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e072bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630662587, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=424us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.663048] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92729334e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C611-0-0, dag_ret:0, dag_status:2, start_time:1727337630662533, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287d2f5d0, output_block_cnt_arr:0x7f9287d2f5d8, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337630662587, estimated_finish_time:1727337647663048}) [2024-09-26 16:00:30.663054] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f92729338a0, type:15, status:2, dag:{ObIDag:{this:0x7f92729334e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C611-0-0, dag_ret:0, dag_status:2, start_time:1727337630662533, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663060] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=6] task finish process(ret=0, start_time=1727337630662586, end_time=1727337630663060, runtime=474, *this={this:0x7f92729338a0, type:15, status:2, dag:{ObIDag:{this:0x7f92729334e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C611-0-0, dag_ret:0, dag_status:2, start_time:1727337630662533, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663118] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=2] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.663125] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337630663123, task_id:YB42C0A8002D-0006230056D8C613-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.663128] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=3] schedule one task(task=0x7f9272934300, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.663133] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337630663132, task_id:YB42C0A8002D-0006230056D8C614-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.663136] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=2] schedule one task(task=0x7f9272934830, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.663141] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337630663140, task_id:YB42C0A8002D-0006230056D8C615-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.663143] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=2] schedule one task(task=0x7f9272934d60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.663164] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.663168] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.663168] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.663171] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:30.663171] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.663174] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:30.663175] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.663178] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.663179] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.663181] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.663186] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.663187] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.663180] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=328, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:26, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:28}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:5, allocator:0x7f926919e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:28, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,]}) [2024-09-26 16:00:30.663208] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.663211] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:30.663212] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:30.663215] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:30.663217] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:30.663222] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.663302] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.663306] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.663337] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337630622073, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.663380] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630663155, end_time=1727337630663379, runtime=224, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f92729334e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C611-0-0, dag_ret:0, dag_status:2, start_time:1727337630662533, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663389] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=856, dag_cnt=5, dag_cnts_[dag.get_type()]=5, &dag=0x7f92729334e0, dag={ObIDag:{this:0x7f92729334e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C611-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630662533, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.663386] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337630622107, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.663397] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=10] update column cnt(column_cnt_=29, store_column_cnt_=29, column_cnt_=29, column_array_.count()=0) [2024-09-26 16:00:30.663426] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C611-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630662531, task_id:YB42C0A8002D-0006230056D8C611-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.663400] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9266505490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.663435] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=30] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.663438] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.663443] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337630622140, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.663444] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:329}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.663452] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.663457] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337630622090, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.663479] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=8] update column cnt(column_cnt_=12, store_column_cnt_=12, column_cnt_=12, column_array_.count()=0) [2024-09-26 16:00:30.663481] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.663484] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.663482] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9263a45490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.663477] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:329}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9266505490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:329}, table_count:1, [{i:0, table_key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e089f20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630663158, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=284us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.663501] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.663490] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101067}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.663504] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.663503] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272933f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C613-0-0, dag_ret:0, dag_status:2, start_time:1727337630663128, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665055f0, output_block_cnt_arr:0x7f92665055f8, concurrent_cnt:1, estimate_row_cnt:62, estimate_occupy_size:0, latest_update_ts:1727337630663158, estimated_finish_time:1727337647663503}) [2024-09-26 16:00:30.663509] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272934300, type:15, status:2, dag:{ObIDag:{this:0x7f9272933f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C613-0-0, dag_ret:0, dag_status:2, start_time:1727337630663128, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663514] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=5] task finish process(ret=0, start_time=1727337630663156, end_time=1727337630663514, runtime=358, *this={this:0x7f9272934300, type:15, status:2, dag:{ObIDag:{this:0x7f9272933f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C613-0-0, dag_ret:0, dag_status:2, start_time:1727337630663128, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663509] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:330}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.663509] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101067}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101067}, table_count:1, [{i:0, table_key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b86a2d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630663164, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=324us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.663535] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92729349a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C615-0-0, dag_ret:0, dag_status:2, start_time:1727337630663143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728fb4a0, output_block_cnt_arr:0x7f92728fb4a8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337630663164, estimated_finish_time:1727337647663535}) [2024-09-26 16:00:30.663526] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:330}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9263a45490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:330}, table_count:1, [{i:0, table_key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e107fe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630662549, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=958us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.663540] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272934d60, type:15, status:2, dag:{ObIDag:{this:0x7f92729349a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C615-0-0, dag_ret:0, dag_status:2, start_time:1727337630663143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663545] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=5] task finish process(ret=0, start_time=1727337630663162, end_time=1727337630663545, runtime=383, *this={this:0x7f9272934d60, type:15, status:2, dag:{ObIDag:{this:0x7f92729349a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C615-0-0, dag_ret:0, dag_status:2, start_time:1727337630663143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663553] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272933a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C612-0-0, dag_ret:0, dag_status:2, start_time:1727337630662538, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a455f0, output_block_cnt_arr:0x7f9263a455f8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337630662549, estimated_finish_time:1727337647663553}) [2024-09-26 16:00:30.663559] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272933dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9272933a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C612-0-0, dag_ret:0, dag_status:2, start_time:1727337630662538, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663564] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=4] task finish process(ret=0, start_time=1727337630662548, end_time=1727337630663563, runtime=1015, *this={this:0x7f9272933dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9272933a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C612-0-0, dag_ret:0, dag_status:2, start_time:1727337630662538, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663596] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337630622055, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.663604] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=15, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.663612] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.663615] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.663618] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:452}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:17, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.663633] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:452}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:452}, table_count:1, [{i:0, table_key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8652d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630662912, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=705us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.663648] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=12] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272932fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C610-0-0, dag_ret:0, dag_status:2, start_time:1727337630662528, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d480, output_block_cnt_arr:0x7f9264b1d488, concurrent_cnt:1, estimate_row_cnt:10, estimate_occupy_size:0, latest_update_ts:1727337630662912, estimated_finish_time:1727337647663648}) [2024-09-26 16:00:30.663653] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272933370, type:15, status:2, dag:{ObIDag:{this:0x7f9272932fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C610-0-0, dag_ret:0, dag_status:2, start_time:1727337630662528, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663657] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=4] task finish process(ret=0, start_time=1727337630662910, end_time=1727337630663657, runtime=747, *this={this:0x7f9272933370, type:15, status:2, dag:{ObIDag:{this:0x7f9272932fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C610-0-0, dag_ret:0, dag_status:2, start_time:1727337630662528, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663668] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=2] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.663672] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=3] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.663675] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=2] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.663678] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=1] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.663723] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=330, range=0x7f92aa352200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a66060, count:7, allocator:0x7f9269ffa7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9264a663b0, local_datums:0x7f9264a663b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:30.663769] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=329, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685ea060, count:6, allocator:0x7f926366c7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f92685ea390, local_datums:0x7f92685ea390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:30.663896] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=27][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:30.663927] ERROR issue_dba_error (ob_log.cpp:1875) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.663932] ERROR issue_dba_error (ob_log.cpp:1875) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.663931] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.663934] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.663940] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=0] partition merge iter row count(i=0, row_count=47, ghost_row_count=0, pkey={tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337630622107, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.663945] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=0] partition merge iter row count(i=0, row_count=141, ghost_row_count=0, pkey={tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337630622090, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.663958] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630663730, end_time=1727337630663958, runtime=228, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9272933f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C613-0-0, dag_ret:0, dag_status:2, start_time:1727337630663128, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663963] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630663681, end_time=1727337630663963, runtime=282, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272933a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C612-0-0, dag_ret:0, dag_status:2, start_time:1727337630662538, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.663966] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=838, dag_cnt=4, dag_cnts_[dag.get_type()]=4, &dag=0x7f9272933f40, dag={ObIDag:{this:0x7f9272933f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C613-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630663128, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.663973] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C613-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630663123, task_id:YB42C0A8002D-0006230056D8C613-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.663976] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337630622123, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:30.663983] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=25, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:30.664003] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=17] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:30.664006] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:30.664011] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:332}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:27, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:30.664028] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:332}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:332}, table_count:1, [{i:0, table_key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8648b00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337630663204, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=805us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:30.664048] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272934470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C614-0-0, dag_ret:0, dag_status:2, start_time:1727337630663135, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3490, output_block_cnt_arr:0x7f9287db3498, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337630663204, estimated_finish_time:1727337647664048}) [2024-09-26 16:00:30.664054] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272934830, type:15, status:2, dag:{ObIDag:{this:0x7f9272934470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C614-0-0, dag_ret:0, dag_status:2, start_time:1727337630663135, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.664059] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=4] task finish process(ret=0, start_time=1727337630663203, end_time=1727337630664058, runtime=855, *this={this:0x7f9272934830, type:15, status:2, dag:{ObIDag:{this:0x7f9272934470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C614-0-0, dag_ret:0, dag_status:2, start_time:1727337630663135, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.664075] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1537, dag_cnt=3, dag_cnts_[dag.get_type()]=3, &dag=0x7f9272933a10, dag={ObIDag:{this:0x7f9272933a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C612-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630662538, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.664082] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C612-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630662536, task_id:YB42C0A8002D-0006230056D8C612-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.664126] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101067, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d2e060, count:8, allocator:0x7f92a31f67d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e3e0, local_datums:0x7f9287d2e3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:30.664212] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=1] schedule one task(task=0x7f9264ae8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:30.664259] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=452, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:15, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:17}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:5, allocator:0x7f926912a7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:17, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,]}) [2024-09-26 16:00:30.664327] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.664329] ERROR issue_dba_error (ob_log.cpp:1875) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.664330] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.664332] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.664337] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=0] partition merge iter row count(i=0, row_count=10, ghost_row_count=0, pkey={tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337630622055, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.664341] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=0] partition merge iter row count(i=0, row_count=227, ghost_row_count=0, pkey={tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337630622140, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.664354] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630664229, end_time=1727337630664354, runtime=125, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9272932fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C610-0-0, dag_ret:0, dag_status:2, start_time:1727337630662528, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.664358] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630664101, end_time=1727337630664358, runtime=257, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f92729349a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C615-0-0, dag_ret:0, dag_status:2, start_time:1727337630663143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.664363] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1834, dag_cnt=2, dag_cnts_[dag.get_type()]=2, &dag=0x7f9272932fb0, dag={ObIDag:{this:0x7f9272932fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C610-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630662528, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.664369] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C610-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630662524, task_id:YB42C0A8002D-0006230056D8C610-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.664382] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1239, dag_cnt=1, dag_cnts_[dag.get_type()]=1, &dag=0x7f92729349a0, dag={ObIDag:{this:0x7f92729349a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C615-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630663143, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.664389] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C615-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337630663140, task_id:YB42C0A8002D-0006230056D8C615-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.664386] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=332, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:25, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:27}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264b1c060, count:6, allocator:0x7f926912a7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:27, datum_buffer:{capacity:32, datums:0x7f9264b1c390, local_datums:0x7f9264b1c390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,]}) [2024-09-26 16:00:30.664493] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:30.664497] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:30.664507] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337630622123, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:30.664529] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337630664353, end_time=1727337630664529, runtime=176, *this={this:0x7f9264ae8030, type:1, status:2, dag:{ObIDag:{this:0x7f9272934470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C614-0-0, dag_ret:0, dag_status:2, start_time:1727337630663135, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:30.664541] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1406, dag_cnt=0, dag_cnts_[dag.get_type()]=0, &dag=0x7f9272934470, dag={ObIDag:{this:0x7f9272934470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C614-0-0, dag_ret:-4184, dag_status:5, start_time:1727337630663135, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:30.664549] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C614-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337630663132, task_id:YB42C0A8002D-0006230056D8C614-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:30.673647] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:30.673670] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.679566] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855DB-0-0] [lt=15] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:30.703950] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:30.703962] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.708828] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337630708827, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007741, add_bytes: 0 [2024-09-26 16:00:30.733217] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=5] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:30.734202] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:30.734210] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.736178] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:30.750851] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:30.750862] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337630750847) [2024-09-26 16:00:30.750868] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337630650264, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:30.750884] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223359987308, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.750893] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:30.750902] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:30.750907] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:30.750910] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:30.750917] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.750919] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.750921] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337630750871) [2024-09-26 16:00:30.753773] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=35] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337630296333]}) [2024-09-26 16:00:30.757806] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:30.757910] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:30.758514] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=1226823, clean_num=31457) [2024-09-26 16:00:30.758582] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=24] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:30.765493] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=51] success to notify_need_writing_throttling True [2024-09-26 16:00:30.765500] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.795780] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:30.795788] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.796832] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:30.796843] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337630796830], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:30.796858] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337630796830], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:30.796862] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:30.796867] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:30.821142] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.821154] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.821159] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.821164] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.821173] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.821170] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.821180] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.821183] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=11] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.822097] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.822115] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.822123] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:30.825058] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337630796830]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:30.825067] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=9][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:30.827024] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:30.827046] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.851465] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:30.851476] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337630851460) [2024-09-26 16:00:30.851482] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337630750871, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:30.851488] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:30.851497] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:30.851500] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:30.851834] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:30.851846] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:30.851851] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:30.851854] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:30.851858] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:30.851860] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:30.851863] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:30.851865] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:30.851867] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:30.851869] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:30.851871] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:30.851873] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:30.851876] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:30.851877] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:30.851884] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:30.851886] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:30.851889] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:30.851895] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=6][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:30.851898] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:30.851901] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:30.851903] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:30.851921] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=16][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:30.851928] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:30.851931] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:30.851936] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:30.851942] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:30.851945] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:30.851948] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573C-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:30.851954] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:30.851957] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:30.851959] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:30.851962] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337630851758, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:30.851966] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:30.851968] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:30.851997] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:30.852000] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337630852000, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=507, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:30.852007] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:30.852010] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:30.852027] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223360087939, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.852031] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:30.852037] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:30.852042] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:30.852044] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:30.852048] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.852050] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.852053] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337630852014) [2024-09-26 16:00:30.854372] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337630796830]}) [2024-09-26 16:00:30.857283] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:00:30.857290] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.859249] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337631789608, now=[mts=1727337629859807]) [2024-09-26 16:00:30.859259] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=9][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337631789608, now=[mts=1727337629859807], now0=[mts=1727337629859807], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:30.859266] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a3121f40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500690, assoc_session_id:3221500690, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337629858178, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:30.859302] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=35][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a2319bf8, id:3221500690, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3121f40}) [2024-09-26 16:00:30.859309] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500690, *tx_desc={this:0x7f92a3121f40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500690, assoc_session_id:3221500690, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337629858178, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:30.859329] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=19] start stmt(ret=-4283, auto_commit=true, session_id=3221500690, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3121f40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500690, assoc_session_id:3221500690, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337629858178, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337629859785, use_das=false, nested_level=0, session={this:0x7f92a2319bf8, id:3221500690, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3121f40}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:30.859348] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=18][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:30.859354] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:30.859356] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:30.859359] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:30.859364] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:30.859367] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:30.859371] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=1000237, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337631859608) [2024-09-26 16:00:30.859375] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:30.859382] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:30.859384] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:30.859385] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:30.859390] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:30.859422] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:30.859425] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:30.887537] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:30.887546] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.889529] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=4] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-09-26 16:00:30.889537] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=6] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=686) [2024-09-26 16:00:30.906124] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337629906928]) [2024-09-26 16:00:30.906134] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337629906928], now0=[mts=1727337629906928], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:30.906140] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337629906023, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:30.906176] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=35][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:30.906183] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337629906023, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:30.906191] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337629906023, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:30.906207] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:30.906211] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:30.906213] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:30.906217] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:30.906219] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:30.906222] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=26, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:30.906226] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] will sleep(sleep_us=26000, remain_us=2645602, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:30.917801] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:30.917810] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.917821] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:30.923975] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2024-09-26 16:00:30.923992] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=15][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2024-09-26 16:00:30.924005] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2024-09-26 16:00:30.924010] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2024-09-26 16:00:30.924013] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:30.924016] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:30.924020] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2024-09-26 16:00:30.924023] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:30.924025] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:30.924027] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:30.924029] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:30.924031] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:30.924033] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:30.924035] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:30.924042] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:30.924044] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:30.924047] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:30.924054] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=6][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:30.924057] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", ret=-5019) [2024-09-26 16:00:30.924059] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:30.924062] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:30.924067] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:30.924073] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:30.924076] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:30.924077] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:30.924084] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:30.924087] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:30.924090] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C41-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, aret=-5019, ret=-5019) [2024-09-26 16:00:30.924093] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-09-26 16:00:30.924096] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:30.924099] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:30.924102] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337630923905, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-09-26 16:00:30.924105] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:30.924108] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:945) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] failed to read(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-09-26 16:00:30.924161] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] fail to inner get ls status info(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2024-09-26 16:00:30.924168] WDIAG [STORAGE.DUP_TABLE] refresh_dup_tablet_schema_ (ob_dup_table_util.cpp:97) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] get duplicate ls status info failed(ret=-5019, dup_ls_status_info={tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2024-09-26 16:00:30.924174] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:148) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] refresh dup table schema failed(ret=-5019) [2024-09-26 16:00:30.924176] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:236) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] scan dup ls to find dup_tablet failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:30.924179] WDIAG [STORAGE.DUP_TABLE] runTimerTask (ob_dup_table_util.cpp:73) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute dup ls scan failed(tmp_ret=-5019) [2024-09-26 16:00:30.933508] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:26, local_retry_times:26, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:30.933566] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=44][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:30.933570] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:30.933572] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:30.933578] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:30.933618] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:30.933624] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:30.936454] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:30.949128] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:30.949137] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.952039] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:30.952047] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:30.952061] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337630952037) [2024-09-26 16:00:30.952066] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337630851485, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:30.952083] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223360187502, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:30.952089] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:30.952104] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:30.952109] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:30.952112] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:30.952118] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.952121] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:30.952123] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337630952068) [2024-09-26 16:00:30.954964] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337630796830]}) [2024-09-26 16:00:30.958982] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:30.959083] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=50, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:30.959705] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1258280, clean_num=31457) [2024-09-26 16:00:30.969663] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:30.980433] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:30.980440] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:30.995204] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221500696, support_ssl=false) [2024-09-26 16:00:30.995229] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221500696, sess.client_addr_="127.0.0.1:36322") [2024-09-26 16:00:30.995235] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500696, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:30.995254] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:30.995262] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2688][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:30.995310] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9C-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:30.995514] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9C-0-0] [lt=6][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:30.995520] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9C-0-0] [lt=5][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:""WoLl[b\'J"e[v`]1|@i"}, ret=-4043) [2024-09-26 16:00:30.995526] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9C-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:30.995530] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9C-0-0] [lt=4][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:30.995551] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9C-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:30.995576] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9C-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500696, proxy_sessid:0}) [2024-09-26 16:00:30.995580] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9C-0-0] [lt=3] mark session id unused(sessid=3221500696) [2024-09-26 16:00:30.995584] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9C-0-0] [lt=2][errcode=0] server close connection(sessid=3221500696, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:30.995587] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9C-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500696, proxy_sessid=0) [2024-09-26 16:00:30.995589] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9C-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:30.995592] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9C-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500696, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:30.995612] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500696, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337630994996, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:30.995620] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500696, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:30.995624] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500696, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337630994996, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:30.995628] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500696, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:31.004782] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: SHARED_BLOCK_ReaderWriter_IO, mode: read, size: 4096.00, iops: 52.68, rt: 3110.21) [2024-09-26 16:00:31.004792] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:31.004811] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10841.00) [2024-09-26 16:00:31.004814] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:31.004820] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:31.004824] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:31.010627] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:31.010636] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.023454] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F6-0-0] [lt=37][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:31.023476] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F6-0-0] [lt=21][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:31.023485] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F6-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:31.023491] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F6-0-0] [lt=6][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:31.034416] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:31.034428] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:31.034432] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:31.034446] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:31.034450] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:31.034452] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:31.034455] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:31.034458] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:31.034460] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:31.034462] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:31.034464] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:31.034466] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:31.034469] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:31.034470] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:31.034478] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:31.034481] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:31.034485] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:31.034489] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:31.034504] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:31.034508] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:31.034511] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:31.034517] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:31.034523] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:31.034525] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:31.034526] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:31.034532] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:31.034535] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85878-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:31.034539] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85878-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:31.034542] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85878-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:31.034545] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85878-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:31.034548] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:31.034552] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85878-0-0] [lt=4][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337631034347, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:31.034556] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85878-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:31.034558] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85878-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:31.034581] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85878-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:31.034584] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85878-0-0] [lt=2][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:31.040933] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:31.040940] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.042866] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=6] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:31.043162] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=6] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:31.043173] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:31.043177] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:31.043180] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:31.043186] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=5][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:31.043189] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:31.043191] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:31.043194] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:31.043196] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:31.043198] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:31.043201] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:31.043205] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:31.043207] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:31.043209] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:31.043224] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E56-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:31.052685] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:31.052695] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337631052682) [2024-09-26 16:00:31.052713] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337630952068, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:31.052731] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223360289128, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:31.052739] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:31.052745] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:31.052750] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:31.052753] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:31.052761] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.052763] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.052765] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337631052717) [2024-09-26 16:00:31.056555] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337630796830]}) [2024-09-26 16:00:31.056604] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D82F-0-0] [lt=19][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337631056548}) [2024-09-26 16:00:31.072425] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:31.072432] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.079983] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337630796830]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:31.079992] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=9][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:31.080950] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=10] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:31.102476] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.102483] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.132718] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:31.132728] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.137938] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=660576, replace_num=15728) [2024-09-26 16:00:31.137949] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:31.153249] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:31.153261] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337631153246) [2024-09-26 16:00:31.153266] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337631052716, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:31.153291] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223360389760, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:31.153298] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:31.153306] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:31.153312] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:31.153314] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:31.153320] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.153323] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.153324] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337631153269) [2024-09-26 16:00:31.158113] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=58] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337630796830]}) [2024-09-26 16:00:31.160217] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:31.160322] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=56, refresh_score_time=16, wash_time=2) [2024-09-26 16:00:31.160934] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1289737, clean_num=31457) [2024-09-26 16:00:31.163000] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.163012] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.194283] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:00:31.194298] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.201337] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:31.201348] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:31.201352] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:31.224497] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.224506] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.232693] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:31.232707] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:31.232711] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:31.232715] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:31.232718] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:31.232721] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:31.232724] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:31.232726] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:31.232728] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:31.232730] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:31.232732] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:31.232734] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:31.232770] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485887-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:31.253829] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:31.253838] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337631253826) [2024-09-26 16:00:31.253843] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337631153268, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:31.253858] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223360490331, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:31.253864] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:31.253871] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:31.253875] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:31.253878] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:31.253883] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.253885] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.253887] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337631253845) [2024-09-26 16:00:31.254796] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.254805] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.258708] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337630796830]}) [2024-09-26 16:00:31.285079] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:31.285093] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.287052] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337631287051, bytes: 14599012, bw: 0.003616 MB/s, add_ts: 1007972, add_bytes: 3822 [2024-09-26 16:00:31.292918] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:55, tid:2919}]) [2024-09-26 16:00:31.293877] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337632223292, now=[mts=1727337630293589]) [2024-09-26 16:00:31.293886] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337632223292, now=[mts=1727337630293589], now0=[mts=1727337630293589], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:31.293892] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926901ff40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337630292961, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:31.293909] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926901ff40}) [2024-09-26 16:00:31.293927] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=18][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926901ff40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337630292961, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:31.293936] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=8] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926901ff40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337630292961, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337630293564, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926901ff40}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:31.293953] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:31.293957] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:31.293960] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:31.293963] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:31.293966] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:31.293970] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:31.293974] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=999318, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337632293292) [2024-09-26 16:00:31.293978] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:31.293985] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:31.293987] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:31.293988] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:31.293993] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:31.294027] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:00:31.294032] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:31.297877] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.297886] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337631297874], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:31.297892] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337631297874], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:31.297896] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=4955, get_gts_with_stc_cnt=11553, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-09-26 16:00:31.302689] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D372D-0-0] [lt=12][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:31.315419] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:31.315427] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.338810] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:31.345640] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:31.345647] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.354478] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:31.354491] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337631354474) [2024-09-26 16:00:31.354497] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337631253845, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:31.354503] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:31.354513] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:31.354516] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:31.354854] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:31.354865] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:31.354869] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:31.354872] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:31.354875] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:31.354878] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:31.354881] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:31.354884] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:31.354887] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:31.354889] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:31.354891] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:31.354896] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:31.354898] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:31.354900] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:31.354906] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:31.354909] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:31.354911] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:31.354913] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:31.354916] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:31.354918] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:31.354921] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:31.354926] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:31.354932] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:31.354934] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:31.354936] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:31.354942] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:31.354945] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:31.354949] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573D-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:31.354951] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:31.354954] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:31.354957] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:31.354960] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337631354765, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:31.354965] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:31.354967] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:31.354996] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:31.355000] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337631354999, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=491, wlock_time=19, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:31.355024] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:31.355027] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:31.355044] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223360591009, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:31.355051] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:31.355057] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:31.355063] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:31.355066] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:31.355072] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.355074] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.355076] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337631355031) [2024-09-26 16:00:31.359022] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337631297874]}) [2024-09-26 16:00:31.361407] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:31.361531] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=51, refresh_score_time=24, wash_time=2) [2024-09-26 16:00:31.362131] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1321194, clean_num=31457) [2024-09-26 16:00:31.375917] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:31.375925] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.379116] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=15] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:31.379129] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:31.379135] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=5][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:31.379138] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=2][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:31.379141] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:31.379146] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=4] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:31.379148] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:31.379150] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:31.379153] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:31.379155] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:31.379157] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:31.379160] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=3][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:31.379161] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:31.379164] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:31.379168] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:31.379170] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:31.379185] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:31.379189] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:31.379192] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:31.379194] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:31.379196] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-09-26 16:00:31.379201] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:31.379206] WDIAG [SERVER] do_process (obmp_query.cpp:771) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=5][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-09-26 16:00:31.379212] WDIAG [SERVER] do_process (obmp_query.cpp:903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, session={this:0x7f92a230bdf8, id:3221487638, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-09-26 16:00:31.379222] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4B-0-0] [lt=6] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:31.383673] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:31.406193] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:31.406199] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.436459] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:31.436468] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.443277] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2] ====== tenant freeze timer task ====== [2024-09-26 16:00:31.443303] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=22] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:31.443335] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337630620744, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337630620709, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337630620761, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:31.443369] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:31.443415] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D4-0-0] [lt=3] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:31.443424] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D4-0-0] [lt=7] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:31.443431] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D4-0-0] [lt=4] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:31.443434] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D4-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:31.443439] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D4-0-0] [lt=4][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:31.443442] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D4-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:31.443447] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D4-0-0] [lt=4][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:31.443470] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D4-0-0] [lt=23] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:31.443474] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D4-0-0] [lt=3][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:31.443477] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D4-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:31.443482] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D4-0-0] [lt=4][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:31.443516] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851D4-0-0] [lt=7] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:31.446239] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.446248] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.446253] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.447036] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.447044] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.447049] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.447201] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.447207] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.447211] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.447226] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.447230] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.447234] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.447562] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.447566] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.447568] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.447770] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.447774] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.447777] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.447781] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.447782] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.447784] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.448123] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.448134] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.448136] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.448380] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.448385] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.448387] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.448392] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.448394] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.448395] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.449166] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.449169] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.449170] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.449172] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.449172] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.449174] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.449178] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.449180] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.449181] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.449783] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.449789] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.449791] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.449796] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.449798] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.449799] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.449968] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.449971] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.449973] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:31.450300] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.451141] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.451150] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.451143] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.452086] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.452086] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.452093] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.452769] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.452777] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.452813] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.453334] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.453342] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.453354] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.454041] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.454048] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.454287] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.454819] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.454829] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.455056] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.455471] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.455478] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.455482] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:31.455494] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337631455479) [2024-09-26 16:00:31.455499] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337631354500, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:31.455515] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223360691952, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:31.455520] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:31.455527] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:31.455532] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:31.455534] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:31.455540] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.455542] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.455545] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337631455503) [2024-09-26 16:00:31.456000] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.456004] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.456010] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.456693] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.456702] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.456740] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.457952] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.457960] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.457971] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.458958] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.458967] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.459025] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.459035] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337631297874]}) [2024-09-26 16:00:31.459860] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.459891] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.459907] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.460772] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.460782] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.460777] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.461887] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.461896] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.461912] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.462850] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.462858] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.462866] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.463443] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.463472] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.463506] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.464000] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.464010] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.464203] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.464726] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.464750] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.464915] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.465375] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.465381] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.465650] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.466010] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.466019] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.466221] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.466718] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:31.466727] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.466753] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.466760] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.466780] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.467691] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.467703] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.467751] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.468695] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.468706] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.469163] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.469690] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.469711] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.469720] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.470344] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.470365] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.470396] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.471057] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.471058] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.471065] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.471575] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.471577] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.471583] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.472452] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.472460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.472466] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:31.497018] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27] success to notify_need_writing_throttling True [2024-09-26 16:00:31.497024] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.520567] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F7-0-0] [lt=34][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:31.520579] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F7-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:31.520588] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F7-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:31.520593] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F7-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:31.527295] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.527301] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.538880] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:31.539486] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6339, mem_used=10398720) [2024-09-26 16:00:31.556461] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:31.556471] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337631556458) [2024-09-26 16:00:31.556476] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337631455503, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:31.556492] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223360792622, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:31.556498] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:31.556505] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:31.556510] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:31.556513] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:31.556519] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.556522] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.556524] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337631556479) [2024-09-26 16:00:31.557537] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:31.557542] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.559135] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337631297874]}) [2024-09-26 16:00:31.560421] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:00:31.562369] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:31.562495] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=49, refresh_score_time=28, wash_time=2) [2024-09-26 16:00:31.563080] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1352651, clean_num=31457) [2024-09-26 16:00:31.587927] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.587933] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.602502] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.603515] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.603527] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.603535] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.618133] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.618139] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.635681] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:31.648240] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.648250] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.648962] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337631297874]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:31.648975] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=13][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:31.655275] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=3] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:31.656289] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:31.657216] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:31.657229] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337631657213) [2024-09-26 16:00:31.657235] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337631556478, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:31.657253] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223360893716, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:31.657259] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:31.657267] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:31.657273] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:31.657276] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:31.657282] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.657285] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.657287] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337631657239) [2024-09-26 16:00:31.659646] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337631297874]}) [2024-09-26 16:00:31.678693] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:31.678701] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.693629] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4184, dropped:518, tid:2864}]) [2024-09-26 16:00:31.708963] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:31.708969] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.716573] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337631716572, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007745, add_bytes: 0 [2024-09-26 16:00:31.738959] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:31.739253] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.739262] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.757524] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:31.757534] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337631757520) [2024-09-26 16:00:31.757540] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337631657239, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:31.757567] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223360993484, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:31.757575] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:31.757583] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:31.757589] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:31.757592] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:31.757599] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.757602] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.757603] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337631757543) [2024-09-26 16:00:31.759883] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337631297874]}) [2024-09-26 16:00:31.763272] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:31.763402] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=52, refresh_score_time=21, wash_time=2) [2024-09-26 16:00:31.764027] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1384108, clean_num=31457) [2024-09-26 16:00:31.769073] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337631297874]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:31.769083] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:31.769409] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:31.769414] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.798399] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:31.798415] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337631798397], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:31.798421] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337631798397], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:31.798425] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:31.798428] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:31.799490] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.799495] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.803970] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [2915][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=10] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-09-26 16:00:31.821249] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.821257] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.821262] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.821266] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.821502] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.821513] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.821520] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.821526] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.821531] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.821908] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.821916] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:31.829663] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:31.829668] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.857772] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:31.857781] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337631857769) [2024-09-26 16:00:31.857786] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337631757543, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:31.857790] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:31.857799] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:31.857802] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:31.858110] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:31.858121] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:31.858125] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:31.858128] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:31.858132] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:31.858134] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:31.858137] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:31.858139] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:31.858141] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:31.858143] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:31.858145] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:31.858148] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:31.858150] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:31.858152] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:31.858176] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=20][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:31.858179] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:31.858183] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:31.858186] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:31.858189] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:31.858192] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:31.858194] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:31.858200] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:31.858206] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:31.858209] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:31.858210] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:31.858216] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:31.858220] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:31.858222] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573E-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:31.858225] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:31.858228] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:31.858231] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:31.858234] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337631858035, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:31.858238] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:31.858240] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:31.858270] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:31.858275] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337631858274, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=478, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:31.858281] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:31.858284] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:31.858308] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223361094651, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:31.858315] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:31.858321] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:31.858325] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:31.858328] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:31.858334] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.858336] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.858338] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337631858287) [2024-09-26 16:00:31.860351] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337631789608, now=[mts=1727337630859421]) [2024-09-26 16:00:31.860373] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=21][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337631789608, now=[mts=1727337630859421], now0=[mts=1727337630859421], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:31.860381] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:31.860386] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.860379] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a3121f40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500690, assoc_session_id:3221500690, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337630859331, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:31.860395] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a2319bf8, id:3221500690, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3121f40}) [2024-09-26 16:00:31.860403] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500690, *tx_desc={this:0x7f92a3121f40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500690, assoc_session_id:3221500690, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337630859331, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:31.860419] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337631798397]}) [2024-09-26 16:00:31.860422] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=18] start stmt(ret=-4283, auto_commit=true, session_id=3221500690, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3121f40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500690, assoc_session_id:3221500690, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337630859331, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337629859785, use_das=false, nested_level=0, session={this:0x7f92a2319bf8, id:3221500690, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3121f40}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:31.860440] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:31.860447] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:31.860450] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:31.860452] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:31.860464] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:31.860468] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:31.860473] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4] already timeout, do not need sleep(sleep_us=-865, remain_us=-865, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, timeout_timestamp=1727337631859608) [2024-09-26 16:00:31.860477] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:31.860483] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:31.860486] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:31.860487] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:31.860492] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:31.860526] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1764) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=6][errcode=-4012] get gts timeout(ret=-4012, expire_ts=1727337631789608, current_time=1727337631859334) [2024-09-26 16:00:31.860529] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4012] acquire global snapshot fail(ret=-4012, gts_ahead=0, expire_ts=1727337631789608, now=[mts=1727337631860525], now0=[mts=1727337631860525], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:31.860532] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4012] acquire global snapshot fail(ret=-4012, tx={this:0x7f92a3121f40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500690, assoc_session_id:3221500690, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337631859334, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:31.860539] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=7][errcode=-4012] fail to get snapshot(ret=-4012, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a2319bf8, id:3221500690, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3121f40}) [2024-09-26 16:00:31.860543] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=4][errcode=-4012] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4012, session_id=3221500690, *tx_desc={this:0x7f92a3121f40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500690, assoc_session_id:3221500690, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337631859334, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:31.860548] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=5] start stmt(ret=-4012, auto_commit=true, session_id=3221500690, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3121f40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500690, assoc_session_id:3221500690, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337631859334, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337629859785, use_das=false, nested_level=0, session={this:0x7f92a2319bf8, id:3221500690, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3121f40}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:31.860558] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=10][errcode=-4012] fail to start stmt(ret=-4012, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:31.860561] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4012] fail start stmt(ret=-4012) [2024-09-26 16:00:31.860563] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1][errcode=-4012] execute plan failed(ret=-4012) [2024-09-26 16:00:31.860564] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1][errcode=-4012] open result set failed(ret=-4012) [2024-09-26 16:00:31.860566] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:31.860567] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:31.860570] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:31.860573] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:31.860575] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:31.860577] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-09-26 16:00:31.860581] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:31.860584] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=3] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000799) [2024-09-26 16:00:31.860586] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688507F-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:31.860589] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:31.860591] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:31.860593] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:31.860596] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8340, start=1727337629859782, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:31.860599] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:00:31.860601] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:31.860623] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:31.860626] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:31.860629] WDIAG [SHARE] get_snapshot_gc_scn (ob_global_stat_proxy.cpp:164) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:31.860632] WDIAG [STORAGE] get_global_info (ob_tenant_freeze_info_mgr.cpp:811) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to get global info(ret=-4012, tenant_id=1) [2024-09-26 16:00:31.860635] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:954) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] failed to get global info(ret=-4012) [2024-09-26 16:00:31.860637] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1008) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to try update info(tmp_ret=-4012, tmp_ret="OB_TIMEOUT") [2024-09-26 16:00:31.860644] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1044) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4] success to update min reserved snapshot(reserved_snapshot=0, duration=1800, snapshot_gc_ts_=0) [2024-09-26 16:00:31.860647] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1071) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2] update reserved snapshot finished(cost_ts=4, reserved_snapshot=0) [2024-09-26 16:00:31.890711] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.890717] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.921049] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.921056] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.921065] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:31.935144] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337630933617]) [2024-09-26 16:00:31.935154] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337630933617], now0=[mts=1727337630933617], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:31.935160] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337630931588, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:31.935178] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:31.935185] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337630931588, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:31.935192] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337630931588, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:31.935209] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:31.935213] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:31.935216] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:31.935218] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:31.935221] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:31.935224] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=27, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:31.935229] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4] will sleep(sleep_us=27000, remain_us=1616599, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:31.939487] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:31.951577] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:31.951597] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.958040] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:31.958050] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:31.958064] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337631958037) [2024-09-26 16:00:31.958069] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337631857789, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:31.958088] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223361193913, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:31.958094] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:31.958102] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:31.958107] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:31.958109] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:31.958117] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.958120] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:31.958121] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337631958072) [2024-09-26 16:00:31.960662] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337631798397]}) [2024-09-26 16:00:31.962376] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:27, local_retry_times:27, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:31.962389] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:31.962391] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:31.962404] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:31.962410] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:31.962450] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:31.962453] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:31.964158] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:31.964266] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=55, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:31.964942] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1415565, clean_num=31457) [2024-09-26 16:00:31.969783] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] tx gc loop thread is running(MTL_ID()=1) [2024-09-26 16:00:31.969788] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] try gc retain ctx [2024-09-26 16:00:31.970190] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-09-26 16:00:31.970216] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:31.981709] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:31.981718] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:31.996495] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500700, support_ssl=false) [2024-09-26 16:00:31.996517] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221500700, sess.client_addr_="127.0.0.1:36324") [2024-09-26 16:00:31.996524] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500700, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:31.996542] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:31.996551] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:31.996590] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9B-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:31.996765] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9B-0-0] [lt=3][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:31.996776] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9B-0-0] [lt=9][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"uOiBOFO=$91+$o'o|Am#"}, ret=-4043) [2024-09-26 16:00:31.996782] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9B-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:31.996786] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9B-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:31.996793] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9B-0-0] [lt=1] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:31.996815] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9B-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500700, proxy_sessid:0}) [2024-09-26 16:00:31.996819] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9B-0-0] [lt=4] mark session id unused(sessid=3221500700) [2024-09-26 16:00:31.996824] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9B-0-0] [lt=3][errcode=0] server close connection(sessid=3221500700, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:31.996827] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9B-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500700, proxy_sessid=0) [2024-09-26 16:00:31.996830] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9B-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:31.996832] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9B-0-0] [lt=1] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500700, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:31.996863] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500700, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337631995368, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:31.996872] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] kill and revert session(conn.sessid_=3221500700, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:31.996876] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500700, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337631995368, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:31.996880] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500700, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:32.006841] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 9385.00) [2024-09-26 16:00:32.006851] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:32.006863] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:32.006867] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:32.012194] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:32.012201] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.020659] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F8-0-0] [lt=40][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:32.020672] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F8-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:32.020693] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F8-0-0] [lt=20][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:32.020699] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F8-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:32.021784] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=6] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:32.021797] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-09-26 16:00:32.021801] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:32.021804] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-09-26 16:00:32.021808] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:32.021810] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:32.021813] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-09-26 16:00:32.021816] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:32.021819] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:32.021821] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:32.021823] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=1][errcode=-5019] resolve table failed(ret=-5019) [2024-09-26 16:00:32.021825] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:2822) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-09-26 16:00:32.021827] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2676) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] resolve joined table failed(ret=-5019) [2024-09-26 16:00:32.021829] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:32.021831] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:32.021833] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:32.021835] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:32.021842] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:32.021847] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=5][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:32.021850] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:32.021852] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:32.021855] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-09-26 16:00:32.021858] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:32.021861] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:32.021866] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:32.021872] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:32.021875] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:32.021876] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:32.021882] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:32.021886] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9C-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:32.021889] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [3024][BlackListServic][T0][YB42C0A8002D-0006230056284F9C-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-09-26 16:00:32.021892] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:32.021895] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:32.021900] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:32.021903] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337632021678, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:32.021906] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:32.021909] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:222) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:32.021912] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:398) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3] start to print blacklist info [2024-09-26 16:00:32.021937] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3] ls blacklist refresh finish(cost_time=519) [2024-09-26 16:00:32.034876] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:32.034888] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:32.034892] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:32.034895] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:32.034899] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:32.034901] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:32.034903] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:32.034906] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:32.034908] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:32.034910] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:32.034914] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=3][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:32.034916] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:32.034917] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:32.034920] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:32.034925] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:32.034928] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:32.034933] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:32.034936] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:32.034938] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:32.034940] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:32.034942] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:32.034948] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:32.034952] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:32.034955] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:32.034956] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:32.034961] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:32.034965] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85879-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:32.034968] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:32.034971] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85879-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:32.034974] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:32.034976] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85879-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:32.034978] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337632034822, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:32.034982] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85879-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:32.034984] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:32.035003] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85879-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:32.035006] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85879-0-0] [lt=2][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:32.042382] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:32.042390] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.043345] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:32.045610] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:32.045619] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:32.045622] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:32.045625] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:32.045628] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:32.045630] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:32.045632] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:32.045634] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:32.045636] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:32.045638] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:32.045639] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:32.045641] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:32.045643] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:32.045646] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:32.045652] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:32.045654] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:32.045656] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:32.045659] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:32.045661] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:32.045663] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:32.045665] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:32.045670] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:32.045674] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:32.045676] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:32.045678] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:32.045684] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:32.045687] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:32.045690] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2582][RSAsyncTask3][T0][YB42C0A8002D-0006230057684E58-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:32.045692] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:32.045695] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:32.045697] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:32.045699] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337632045570, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:32.045702] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:32.045745] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:32.058179] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:32.058189] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337632058177) [2024-09-26 16:00:32.058194] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337631958072, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:32.058212] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223361294680, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:32.058218] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:32.058225] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:32.058231] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:32.058234] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:32.058240] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.058243] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.058245] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337632058198) [2024-09-26 16:00:32.060822] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337631798397]}) [2024-09-26 16:00:32.060845] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D83B-0-0] [lt=9][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337632060815}) [2024-09-26 16:00:32.072496] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:32.072503] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.080991] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=7] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:32.102585] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:32.102590] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.132816] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:32.132823] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.139511] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:32.158218] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:32.158230] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337632158215) [2024-09-26 16:00:32.158236] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337632058198, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:32.158252] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223361394332, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:32.158259] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:32.158267] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:32.158271] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:32.158274] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:32.158282] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.158285] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.158287] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337632158239) [2024-09-26 16:00:32.160867] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337631798397]}) [2024-09-26 16:00:32.163165] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:32.163173] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.165187] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2248114176, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:32.165300] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=50, refresh_score_time=26, wash_time=2) [2024-09-26 16:00:32.165918] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1447022, clean_num=31457) [2024-09-26 16:00:32.193365] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:32.193373] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.197641] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:00:32.197650] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=7] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:32.197657] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:32.223425] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:32.223432] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.233801] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485888-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:32.233816] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485888-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:32.233820] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485888-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:32.233823] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485888-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:32.233827] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485888-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:32.233829] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485888-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:32.233851] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485888-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:32.253550] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:32.253558] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.258284] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:32.258296] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337632258281) [2024-09-26 16:00:32.258302] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337632158238, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:32.258318] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223361494715, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:32.258323] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:32.258331] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:32.258336] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:32.258339] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:32.258344] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.258346] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.258349] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337632258305) [2024-09-26 16:00:32.261283] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337631798397]}) [2024-09-26 16:00:32.283748] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:32.283757] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.293967] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337632293966, bytes: 14602552, bw: 0.003353 MB/s, add_ts: 1006915, add_bytes: 3540 [2024-09-26 16:00:32.293985] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337632223292, now=[mts=1727337631294025]) [2024-09-26 16:00:32.293992] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337632223292, now=[mts=1727337631294025], now0=[mts=1727337631294025], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:32.293999] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926901ff40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337631293874, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:32.294018] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=18][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926901ff40}) [2024-09-26 16:00:32.294026] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926901ff40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337631293874, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:32.294034] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926901ff40, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337631293874, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337630293564, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926901ff40}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:32.294052] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:32.294057] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:32.294060] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:32.294063] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:32.294066] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:32.294070] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:32.294074] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:32.294082] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=5][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:32.294086] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:32.294087] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:32.294096] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:32.294100] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=3] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000536) [2024-09-26 16:00:32.294103] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:32.294107] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:32.294110] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:32.294121] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=1][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:32.294125] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:32.294131] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:32.294163] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=5][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:32.294168] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:32.294171] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:32.294173] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:32.294177] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=4][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:32.294180] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:32.294289] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDE-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:32.294296] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:32.294299] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:32.294315] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:32.294319] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:32.294610] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:00:32.294619] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=9][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:32.298978] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.298989] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337632298976], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:32.298997] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337632298976], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:32.299001] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337632298976]}) [2024-09-26 16:00:32.309959] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D37A7-0-0] [lt=9][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:32.314553] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:32.314561] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.339953] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:32.344832] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:32.344840] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.358496] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:32.358510] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337632358492) [2024-09-26 16:00:32.358538] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337632258305, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:32.358545] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:32.358555] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:32.358558] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:32.358889] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573F-0-0] [lt=5] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:32.358902] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573F-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:32.358946] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788573F-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:32.358985] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337632358984, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=434, wlock_time=13, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:32.358997] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:32.359016] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223361595026, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:32.359021] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:32.359028] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:32.359033] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:32.359036] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:32.359043] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.359046] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.359047] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337632359002) [2024-09-26 16:00:32.361437] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337632298976]}) [2024-09-26 16:00:32.367420] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:32.367539] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=54, refresh_score_time=29, wash_time=2) [2024-09-26 16:00:32.368160] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1478479, clean_num=31457) [2024-09-26 16:00:32.375123] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:32.375132] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.381999] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=7] current monitor number(seq_=-1) [2024-09-26 16:00:32.383915] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=9] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:32.385843] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=21][errcode=-4018] try free send task failed(ret=-4018) [2024-09-26 16:00:32.395622] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:69, tid:2932}]) [2024-09-26 16:00:32.405420] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:32.405429] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.435600] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:32.435609] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.447365] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=21] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.447391] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.447396] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.447412] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=21] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.447416] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.447418] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.448362] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.448376] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.448378] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.448375] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.448388] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.448393] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.449335] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.449334] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.449340] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.449343] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.449341] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.449345] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.450309] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.450309] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.450312] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.450314] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.450314] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.450316] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.452279] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.452304] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.452305] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=25][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.452307] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.452308] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.452309] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.454225] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.454226] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.454228] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.454229] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.454230] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.454231] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.455172] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.455177] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.455181] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.455184] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.455182] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.455188] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.456149] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=15] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.456161] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.456164] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.456166] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.456169] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.456173] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:32.457138] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.457146] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.459097] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.459101] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.459116] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:32.459146] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337632459113) [2024-09-26 16:00:32.459151] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337632358542, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:32.459167] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223361694592, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:32.459173] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:32.459181] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:32.459186] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:32.459189] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:32.459195] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.459198] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.459199] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337632459154) [2024-09-26 16:00:32.460047] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.460051] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.461071] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.461074] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.462005] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.462015] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337632298976]}) [2024-09-26 16:00:32.462006] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.462991] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.462998] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.463968] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=41][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.463971] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.464938] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.464973] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.465944] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:32.465944] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=24][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.465952] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.465950] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.466911] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.466916] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.467889] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.467892] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.468858] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.468914] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.469850] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.469853] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.470817] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.470875] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.471772] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.471788] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.472766] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.472807] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.473740] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.473743] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.474718] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.474725] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.475687] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.475690] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.476665] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.476679] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.477661] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.477670] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.478619] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.478630] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.479608] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.479619] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.480573] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.480574] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.481577] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.481577] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.482533] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.482527] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.484488] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.484489] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.485445] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.485469] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.486457] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.486458] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=30][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.486470] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:262) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=5] ObArchiveSender is running(thread_index=0) [2024-09-26 16:00:32.487417] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.487439] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.488424] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.488427] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.489378] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.489401] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.490336] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.490343] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.491325] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.491333] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.493289] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.493300] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.494260] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.494262] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.495237] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.495252] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.496198] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:32.496208] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.496204] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.496214] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.498162] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.498182] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.499153] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.499154] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.500094] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.501116] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.501133] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.502062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:32.520836] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F9-0-0] [lt=37][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:32.520844] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F9-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:32.520852] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F9-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:32.520857] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858F9-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:32.527563] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:32.527570] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.540134] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6349, mem_used=12478464) [2024-09-26 16:00:32.540149] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:32.557724] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:32.557743] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.559703] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:32.559716] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337632559699) [2024-09-26 16:00:32.559722] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337632459154, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:32.559750] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=223361796209, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:32.559757] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:32.559765] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:32.559771] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:32.559774] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:32.559780] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.559783] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.559785] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337632559734) [2024-09-26 16:00:32.561662] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=4] gc stale ls task succ [2024-09-26 16:00:32.562627] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337632298976]}) [2024-09-26 16:00:32.568604] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:32.568705] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=48, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:32.569311] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1509936, clean_num=31457) [2024-09-26 16:00:32.588030] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:32.588039] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.595162] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:32.595176] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:32.595180] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:32.595184] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:32.595187] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:32.595189] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:32.595192] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:32.595194] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:32.595196] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:32.595198] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:32.595200] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:32.595202] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:32.595205] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:32.595207] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:32.595239] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:32.595243] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:32.595246] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:32.595248] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:32.595250] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:32.595253] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:32.595255] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:32.595261] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:32.595267] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:32.595269] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:32.595271] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:32.595277] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:32.595280] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD1-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:32.595289] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2458][ServerGTimer][T0][YB42C0A8002D-0006230056A84DD1-0-0] [lt=8][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:32.595292] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:32.595295] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:32.595297] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:32.595300] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337632595096, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:32.595303] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:32.595338] WDIAG [SHARE] refresh (ob_all_server_tracer.cpp:568) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] fail to get servers_info(ret=-5019, ret="OB_TABLE_NOT_EXIST", GCTX.sql_proxy_=0x55c7486d2e88) [2024-09-26 16:00:32.595341] WDIAG [SHARE] runTimerTask (ob_all_server_tracer.cpp:626) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] fail to refresh all server map(ret=-5019) [2024-09-26 16:00:32.601727] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.602657] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.602673] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.602675] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.618297] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:32.618304] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.637950] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:32.648614] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:32.648621] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.655360] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:32.658892] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=43] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:32.660296] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:32.660307] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337632660293) [2024-09-26 16:00:32.660312] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337632559734, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:32.660339] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223361896059, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:32.660346] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:32.660353] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:32.660359] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:32.660361] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:32.660367] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.660369] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.660371] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337632660315) [2024-09-26 16:00:32.663214] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337632298976]}) [2024-09-26 16:00:32.678853] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:32.678863] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.709170] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27] success to notify_need_writing_throttling True [2024-09-26 16:00:32.709179] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.723938] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=7] PNIO [ratelimit] time: 1727337632723937, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007365, add_bytes: 0 [2024-09-26 16:00:32.739383] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:32.739392] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.740412] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:32.760890] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:32.760902] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337632760887) [2024-09-26 16:00:32.760908] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337632660315, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:32.760925] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223361996400, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:32.760931] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:32.760940] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:32.760944] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:32.760947] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:32.760952] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.760954] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.760956] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337632760911) [2024-09-26 16:00:32.763848] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337632298976]}) [2024-09-26 16:00:32.769866] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:32.769914] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:32.769919] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.769984] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=53, refresh_score_time=27, wash_time=2) [2024-09-26 16:00:32.770603] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1541393, clean_num=31457) [2024-09-26 16:00:32.799968] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=14] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:32.800036] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:32.800042] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.800097] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:32.800110] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337632800095], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:32.800119] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337632800095], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:32.800124] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:32.800129] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:32.820478] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.820494] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.820502] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.820507] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.821460] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.821471] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.821478] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.821484] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.821489] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.822471] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.822489] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:32.830295] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:32.830301] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.860540] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:32.860545] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.861478] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:32.861489] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337632861474) [2024-09-26 16:00:32.861495] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337632760911, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:32.861502] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:32.861512] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:32.861516] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:32.861942] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:32.861952] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:32.861957] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:32.861962] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:32.861966] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:32.861970] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:32.861974] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:32.861979] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:32.861982] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:32.861985] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:32.861988] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:32.861991] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:32.861994] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:32.861997] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:32.862004] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:32.862008] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:32.862013] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:32.862016] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:32.862020] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:32.862023] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:32.862026] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:32.862034] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=6][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:32.862041] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:32.862046] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:32.862048] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:32.862056] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:32.862060] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:32.862063] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885740-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:32.862067] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:32.862070] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:32.862074] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:32.862078] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337632861849, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:32.862082] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:32.862086] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:32.862120] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:32.862124] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337632862123, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=615, wlock_time=11, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:32.862133] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:32.862138] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:32.862156] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223362098113, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:32.862162] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:32.862171] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:32.862177] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:32.862180] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:32.862186] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.862188] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.862191] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337632862143) [2024-09-26 16:00:32.864432] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337632800095]}) [2024-09-26 16:00:32.871248] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=20][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337632800095]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:32.871260] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:32.891802] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:32.891810] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.923042] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:32.923069] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.923078] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:32.940609] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:32.953314] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:32.953320] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.961548] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:32.961581] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:32.961594] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337632961539) [2024-09-26 16:00:32.961599] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337632861499, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:32.961618] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223362197593, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:32.961626] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:32.961633] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:32.961638] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:32.961641] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:32.961648] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.961650] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:32.961651] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337632961602) [2024-09-26 16:00:32.963730] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337631962449]) [2024-09-26 16:00:32.963741] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337631962449], now0=[mts=1727337631962449], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:32.963748] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337631962146, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:32.963765] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:32.963773] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337631962146, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:32.963781] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337631962146, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:32.963800] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=19][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:32.963817] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:32.963819] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:32.963822] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:32.963825] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:32.963829] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=28, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:32.963833] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] will sleep(sleep_us=28000, remain_us=587995, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4283, timeout_timestamp=1727337633551828) [2024-09-26 16:00:32.964861] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=31] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337632800095]}) [2024-09-26 16:00:32.970474] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:32.970999] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:32.971107] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash time detail, (compute_wash_size_time=87, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:32.971117] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1572850, clean_num=31457) [2024-09-26 16:00:32.983571] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:32.983582] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:32.992372] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:28, local_retry_times:28, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:32.992412] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=34][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:32.992415] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:32.992417] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:32.992434] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:32.992477] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=73) [2024-09-26 16:00:32.992482] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:32.997420] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500704, support_ssl=false) [2024-09-26 16:00:32.997464] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221500704, sess.client_addr_="127.0.0.1:36326") [2024-09-26 16:00:32.997473] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] accept one succ(*s={this:0x7f9260a04030, session_id:3221500704, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:32.997499] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:32.997508] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:32.997545] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9C-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:32.997716] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9C-0-0] [lt=5][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:32.997732] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9C-0-0] [lt=15][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"Jla!!}vZ@p6ZC}m,1P#X"}, ret=-4043) [2024-09-26 16:00:32.997739] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9C-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:32.997743] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9C-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:32.997749] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9C-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:32.997772] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9C-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500704, proxy_sessid:0}) [2024-09-26 16:00:32.997777] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9C-0-0] [lt=4] mark session id unused(sessid=3221500704) [2024-09-26 16:00:32.997781] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9C-0-0] [lt=2][errcode=0] server close connection(sessid=3221500704, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:32.997784] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9C-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500704, proxy_sessid=0) [2024-09-26 16:00:32.997787] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9C-0-0] [lt=3][errcode=-4016] session is null [2024-09-26 16:00:32.997792] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9C-0-0] [lt=5] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500704, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:32.997821] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f9260a04030, session_id:3221500704, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337632997355, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:32.997831] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] kill and revert session(conn.sessid_=3221500704, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:32.997834] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9260a04030, session_id:3221500704, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337632997355, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:32.997838] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500704, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:33.012904] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10756.00) [2024-09-26 16:00:33.012913] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:33.012937] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:33.012942] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:33.013845] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:33.013854] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.029622] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FA-0-0] [lt=32][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:33.029633] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FA-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:33.029642] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FA-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:33.029647] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FA-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:33.035719] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:33.035731] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:33.035735] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:33.035738] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:33.035741] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:33.035743] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:33.035749] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:33.035752] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:33.035756] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:33.035759] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=3][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:33.035761] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:33.035763] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:33.035766] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:33.035768] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:33.035774] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:33.035776] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.035780] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.035784] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:33.035787] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:33.035790] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:33.035793] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:33.035799] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:33.035804] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.035807] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.035808] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:33.035815] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:33.035819] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587A-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:33.035822] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:33.035825] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:33.035828] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:33.035832] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587A-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:33.035835] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337633035649, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:33.035838] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587A-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:33.035840] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587A-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:33.035863] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587A-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:33.035866] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587A-0-0] [lt=3][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:33.045130] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.045139] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.046105] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=6] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:33.046409] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:33.046420] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:33.046424] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:33.046439] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:33.046443] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:33.046446] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:33.046448] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:33.046450] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:33.046455] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:33.046458] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:33.046459] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:33.046461] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:33.046477] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=15][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:33.046480] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:33.046486] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:33.046488] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.046491] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.046496] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:33.046498] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:33.046501] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:33.046503] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:33.046508] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:33.046513] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.046515] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.046517] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:33.046522] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:33.046525] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5A-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:33.046530] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2582][RSAsyncTask3][T0][YB42C0A8002D-0006230057684E5A-0-0] [lt=4][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:33.046532] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:33.046535] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:33.046539] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:33.046542] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337633046364, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:33.046545] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:33.046591] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:33.061726] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:33.061736] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337633061722) [2024-09-26 16:00:33.061741] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337632961602, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:33.061759] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223362297247, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:33.061765] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:33.061771] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:33.061776] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:33.061779] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:33.061784] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.061786] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.061788] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337633061744) [2024-09-26 16:00:33.065626] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337632800095]}) [2024-09-26 16:00:33.065635] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D847-0-0] [lt=9][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337633065621}) [2024-09-26 16:00:33.075401] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.075409] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.081258] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=6] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:33.105679] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:33.105684] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.136970] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:33.136985] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.140814] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:33.162317] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:33.162330] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337633162314) [2024-09-26 16:00:33.162336] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337633061744, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:33.162352] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223362397808, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:33.162357] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:33.162365] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:33.162370] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:33.162375] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:33.162381] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.162383] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.162385] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337633162339) [2024-09-26 16:00:33.166267] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337632800095]}) [2024-09-26 16:00:33.167192] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.167202] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.171264] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:33.171365] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=49, refresh_score_time=13, wash_time=2) [2024-09-26 16:00:33.171982] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=0, clean_num=31457) [2024-09-26 16:00:33.197477] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:33.197493] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.201621] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:33.201631] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:33.201635] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:33.227576] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.227581] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.234368] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:33.234381] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:33.234386] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:33.234390] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:33.234393] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:33.234395] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:33.234399] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:33.234401] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:33.234403] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:33.234405] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:33.234407] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:33.234409] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:33.234411] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:33.234413] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:33.234419] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:33.234422] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.234424] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.234426] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:33.234428] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:33.234431] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:33.234437] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:33.234442] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:33.234447] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.234450] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.234454] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:33.234476] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:33.234481] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=4][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:33.234484] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:33.234487] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:33.234490] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:33.234495] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:33.234498] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337633234304, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:33.234501] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:33.234503] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:33.234544] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=4][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:33.234551] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=7][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:33.234554] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=3][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:33.234556] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:33.234559] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:33.234563] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=3][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:33.234565] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485889-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:33.258018] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:33.258026] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.262375] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:33.262388] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337633262372) [2024-09-26 16:00:33.262393] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337633162339, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:33.262411] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223362498405, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:33.262418] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:33.262425] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:33.262430] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:33.262433] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:33.262439] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.262441] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.262443] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337633262396) [2024-09-26 16:00:33.266569] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337632800095]}) [2024-09-26 16:00:33.273506] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:218) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=29] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-09-26 16:00:33.273517] INFO [COMMON] handle (memory_dump.cpp:498) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-09-26 16:00:33.273544] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3] update_check_range(min_check_version=340, max_check_version=341, global_version=342) [2024-09-26 16:00:33.276034] INFO handle (memory_dump.cpp:626) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3] statistics: tenant_cnt: 3, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 79 198 0 1 2 362 1 0 1 4 1 1 0 1 5 1 5 0 1 7 1 2 0 1 8 60 1 0 1 12 4 2 0 1 13 2 1 0 1 14 1 1 0 1 18 54 3 0 500 0 52 216 0 500 7 2 3 0 500 8 42 1 0 500 9 2 1 0 500 10 6 2 0 500 16 1 1 0 500 17 6 7 0 500 22 3 49 0 500 23 23 10 0 508 0 2 8 0 508 8 3 1 0 cost_time: 2494 [2024-09-26 16:00:33.288156] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.288162] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.295138] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337634224325, now=[mts=1727337632294609]) [2024-09-26 16:00:33.295146] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337634224325, now=[mts=1727337632294609], now0=[mts=1727337632294609], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:33.295152] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269011240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337632293982, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:33.295181] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=28][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269011240}) [2024-09-26 16:00:33.295189] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269011240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337632293982, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:33.295197] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269011240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337632293982, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337632294578, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269011240}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:33.295214] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:33.295219] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:33.295222] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:33.295225] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:33.295228] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:33.295231] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:33.295236] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=999089, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337634294325) [2024-09-26 16:00:33.295240] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:33.295247] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:33.295249] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:33.295251] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:33.295256] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:33.295292] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:00:33.295295] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:33.300580] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.300587] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337633300578], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:33.300594] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337633300578], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:33.301983] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO [ratelimit] time: 1727337633301982, bytes: 14606374, bw: 0.003616 MB/s, add_ts: 1008016, add_bytes: 3822 [2024-09-26 16:00:33.310314] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D37ED-0-0] [lt=19][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:33.314697] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD2-0-0] [lt=1][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-09-26 16:00:33.314703] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD2-0-0] [lt=6][errcode=-4002] refresh sever list failed(ret=-4002) [2024-09-26 16:00:33.314705] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD2-0-0] [lt=2][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-09-26 16:00:33.318582] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.318587] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.341352] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=676304, replace_num=15728) [2024-09-26 16:00:33.341367] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=14] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:33.348863] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:33.348872] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.362595] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:33.362609] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337633362592) [2024-09-26 16:00:33.362616] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337633262396, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:33.362622] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:33.362631] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:33.362634] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:33.362948] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885741-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:33.362961] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885741-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:33.362965] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885741-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:33.362969] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885741-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:33.362972] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885741-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:33.362974] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885741-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:33.362978] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885741-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:33.362981] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885741-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:33.363003] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885741-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:33.363036] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337633363035, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=408, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:33.363046] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:33.363078] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223362599053, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:33.363086] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:33.363093] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:33.363098] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:33.363101] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:33.363108] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.363111] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.363112] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337633363050) [2024-09-26 16:00:33.367003] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337633300578]}) [2024-09-26 16:00:33.372414] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:33.372517] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:33.373119] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=31457, clean_num=31457) [2024-09-26 16:00:33.379138] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:33.379154] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.382088] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=7] current monitor number(seq_=-1) [2024-09-26 16:00:33.384120] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=10] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:33.385992] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [2645][Blacklist][T0][Y0-0000000000000000-0-0] [lt=9] blacklist_loop exec finished(cost_time=10, is_enabled=true, send_cnt=0) [2024-09-26 16:00:33.409460] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.409466] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.439727] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.439735] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.441648] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12] LogLoopThread round_cost_time(us)(round_cost_time=9) [2024-09-26 16:00:33.448488] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.448505] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.448510] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.448515] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.448518] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.448520] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.449462] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.449465] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.449467] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.449469] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.449471] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.449472] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.450518] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.450523] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.450526] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.450529] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.450542] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.450544] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.451208] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.451211] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.451213] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.451217] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.451218] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.451222] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.451229] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] ====== tenant freeze timer task ====== [2024-09-26 16:00:33.451238] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:33.451270] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337630620744, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337630620709, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337630620761, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:33.451298] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:33.451347] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D5-0-0] [lt=3] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:33.451368] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D5-0-0] [lt=18] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:33.451374] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D5-0-0] [lt=4] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:33.451377] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D5-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:33.451381] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D5-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:33.451383] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D5-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:33.451386] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D5-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:33.451395] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D5-0-0] [lt=8] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:33.451397] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D5-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:33.451399] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D5-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:33.451402] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D5-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:33.451435] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851D5-0-0] [lt=3] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:33.451728] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.451731] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.451733] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.452415] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.452431] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.452436] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.452442] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.452445] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.452447] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.453403] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.453408] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.453411] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.453415] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.453416] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.453419] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.454361] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.454373] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.454376] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.454380] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.454382] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.454383] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.454961] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.454967] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.454969] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:33.454972] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.456330] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.456336] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.457272] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.457293] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.458304] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.458312] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.458949] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.458957] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.460200] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.460206] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.460781] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.460788] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.461370] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.461374] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.462145] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.462154] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.463132] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:33.463156] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337633463129) [2024-09-26 16:00:33.463161] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337633362619, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:33.463177] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223362697843, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:33.463183] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:33.463191] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:33.463196] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:33.463199] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:33.463205] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.463208] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.463211] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337633463164) [2024-09-26 16:00:33.463237] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.463241] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.465102] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.465116] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.465649] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.465658] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.467067] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.467074] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337633300578]}) [2024-09-26 16:00:33.467086] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.468021] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.468027] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.468681] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.468687] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.469402] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.469409] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.469997] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:33.470002] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.471016] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.471025] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.471978] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.471983] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.472912] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.472917] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.473905] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.473920] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.474895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.474900] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.475851] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.475861] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.476417] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.476425] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.477254] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.477261] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.477800] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.477807] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.478799] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.478806] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.479745] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.479753] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.480740] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.480746] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.481669] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.481676] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.482708] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.482714] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.483668] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.483673] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.484627] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.484631] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.485598] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.485606] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.486424] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.486431] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.487560] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.487565] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.488310] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.488318] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.488871] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.488877] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.489530] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.489536] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.490160] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.490169] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.490722] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.490729] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.491337] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.491342] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.492097] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.492105] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.492681] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:33.500532] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:33.500544] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.523875] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FB-0-0] [lt=34][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:33.523892] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FB-0-0] [lt=16][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:33.523902] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FB-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:33.523908] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FB-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:33.530966] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:33.530974] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.540332] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6357, mem_used=12478464) [2024-09-26 16:00:33.541494] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=4] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:33.561799] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=4] gc stale ls task succ [2024-09-26 16:00:33.561812] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:33.561816] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.563764] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:33.563775] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337633563761) [2024-09-26 16:00:33.563781] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337633463164, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:33.563799] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223362798860, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:33.563806] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:33.563814] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:33.563819] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:33.563822] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:33.563829] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.563832] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.563833] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337633563784) [2024-09-26 16:00:33.567505] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337633300578]}) [2024-09-26 16:00:33.573670] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:33.573769] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:33.574390] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=62914, clean_num=31457) [2024-09-26 16:00:33.593056] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:33.593063] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.600868] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.600883] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.600889] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.601054] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:28, tid:2458}]) [2024-09-26 16:00:33.601532] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.623320] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:33.623337] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.642888] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:33.653625] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.653632] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.655559] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:33.664743] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:33.664755] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:33.664762] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=6][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:33.664765] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:33.664769] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:33.664771] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:33.664776] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:33.664779] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:33.664781] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:33.664784] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:33.664789] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=5][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:33.664792] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:33.664795] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:33.664797] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:33.664804] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:33.664806] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.664810] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.664812] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:33.664815] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, ret=-5019) [2024-09-26 16:00:33.664818] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:33.664823] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:33.664829] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:33.664834] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.664837] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.664839] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:33.664845] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:33.664849] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:33.664852] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD2-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-09-26 16:00:33.664855] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:33.664858] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:33.664860] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:33.664863] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337633664670, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:33.664867] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:33.664870] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f92b4056738, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, columns_str="zone") [2024-09-26 16:00:33.664876] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:33.664905] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:634) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f92b4056738, where_condition="where svr_ip='192.168.0.45' and svr_port=2882", zone_name_holder=) [2024-09-26 16:00:33.664909] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:567) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:33.664913] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:576) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:33.664915] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:33.664921] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:33.664940] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:33.664946] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337633664938) [2024-09-26 16:00:33.664950] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337633563784, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:33.664966] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223362899946, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:33.664971] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:33.664988] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:33.664992] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:33.664995] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:33.665000] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.665002] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.665004] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337633664953) [2024-09-26 16:00:33.668039] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337633300578]}) [2024-09-26 16:00:33.684088] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:33.684096] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.694973] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855DD-0-0] [lt=8] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:33.694998] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855DF-0-0] [lt=9] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:33.714140] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:33.714151] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.731752] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337633731751, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007814, add_bytes: 0 [2024-09-26 16:00:33.733682] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=10] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:33.741721] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:33.744475] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:33.744482] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.765278] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:33.765291] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337633765275) [2024-09-26 16:00:33.765307] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337633664953, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:33.765347] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223363001443, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:33.765355] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:33.765364] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:33.765369] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:33.765373] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:33.765380] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.765382] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.765384] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337633765311) [2024-09-26 16:00:33.768850] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337633300578]}) [2024-09-26 16:00:33.775844] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:33.775959] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:33.776571] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=94371, clean_num=31457) [2024-09-26 16:00:33.776653] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:33.776658] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.801193] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:33.801201] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337633801189], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:33.801208] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337633801189], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:33.801222] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:33.801226] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:33.806939] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.806946] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.821658] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.821661] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.821672] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.821683] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.821687] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.821692] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.821697] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.822118] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.822562] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.822572] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.822579] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:33.836983] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.836992] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.865416] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:33.865438] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337633865402) [2024-09-26 16:00:33.865457] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337633765310, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:33.865463] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:33.865473] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:33.865484] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:33.865817] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:33.865830] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:33.865835] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:33.865838] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:33.865841] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:33.865843] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:33.865848] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:33.865850] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:33.865853] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:33.865855] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:33.865857] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:33.865859] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:33.865862] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:33.865864] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:33.865870] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:33.865873] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.865875] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.865880] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:33.865883] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:33.865886] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:33.865889] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:33.865895] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:33.865901] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.865903] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.865905] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:33.865911] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:33.865914] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:33.865918] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885742-0-0] [lt=4][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:33.865921] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:33.865924] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:33.865926] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:33.865929] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337633865739, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:33.865933] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:33.865936] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:33.865967] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:33.865972] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337633865972, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=504, wlock_time=20, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:33.865980] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:33.865983] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:33.866001] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223363101521, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:33.866007] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:33.866014] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:33.866018] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:33.866021] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:33.866028] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.866030] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.866032] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337633865987) [2024-09-26 16:00:33.867564] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:33.867571] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.869468] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337633801189]}) [2024-09-26 16:00:33.897750] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.897756] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.928040] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:33.928047] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.928056] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:33.928067] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] BlockGCTimerTask success(ret=0, cost_time_us=29, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.942697] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:33.954806] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-09-26 16:00:33.954818] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-09-26 16:00:33.954823] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-09-26 16:00:33.954826] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-09-26 16:00:33.954829] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:33.954831] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:33.954834] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-09-26 16:00:33.954836] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:33.954838] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:33.954840] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:33.954842] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:33.954844] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:33.954846] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:33.954848] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:33.954882] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:33.954886] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.954890] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.954893] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:33.954898] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:33.954902] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:33.954904] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:33.954909] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:33.954915] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.954917] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.954919] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:33.954925] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:33.954928] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:33.954932] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2024-09-26 16:00:33.954935] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:33.954938] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:33.954942] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:33.954945] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337633954745, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:33.954948] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:33.954951] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1111) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:33.954977] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:801) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=3][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:33.954982] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:67) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=4][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:33.954984] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:436) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:33.954986] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:95) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:33.955000] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:109) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=14] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:33.955217] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-09-26 16:00:33.955226] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2024-09-26 16:00:33.955229] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2024-09-26 16:00:33.955231] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2024-09-26 16:00:33.955234] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:33.955236] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:33.955238] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_tenant' doesn't exist [2024-09-26 16:00:33.955240] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:33.955241] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:33.955243] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:33.955245] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:33.955247] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:33.955249] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:33.955251] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:33.955254] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:33.955258] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.955260] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:33.955261] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:33.955263] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:33.955265] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:33.955267] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:33.955271] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:33.955277] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.955279] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:33.955280] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:33.955284] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:33.955286] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:33.955289] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, aret=-5019, ret=-5019) [2024-09-26 16:00:33.955293] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:00:33.955295] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:33.955297] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:33.955299] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337633955191, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:00:33.955301] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:33.955303] WDIAG [SHARE] read_tenants (ob_unit_table_operator.cpp:1174) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] execute sql failed(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:33.955316] WDIAG [SHARE] get_tenants (ob_unit_table_operator.cpp:108) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] fail read tenants(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:33.955318] WDIAG [SHARE] get_tenants (ob_unit_getter.cpp:198) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] ut_operator get_resource_pools failed(ret=-5019) [2024-09-26 16:00:33.955320] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:125) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2][errcode=-5019] get cluster tenants fail(ret=-5019) [2024-09-26 16:00:33.955331] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:325) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=2] cal_all_part_disk_default_percentage succ(data_dir="/root/observer/store/sstable", clog_dir="/root/observer/store/clog", shared_mode=true, data_disk_total_size=99765694464, data_disk_default_percentage=60, clog_disk_total_size=99765694464, clog_disk_default_percentage=30) [2024-09-26 16:00:33.955344] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:361) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=13] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=99765694464, disk_size=15032385536) [2024-09-26 16:00:33.955347] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=3] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=99765694464) [2024-09-26 16:00:33.955351] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=3] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=99765694464, this={dir::"/root/observer/store/clog/log_pool", dir_fd:15, meta_fd:16, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:9388, max_block_id:9581, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2024-09-26 16:00:33.955356] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:133) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD3-0-0] [lt=5] refresh tenant config(tenants=[], ret=-5019) [2024-09-26 16:00:33.958364] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5] success to notify_need_writing_throttling True [2024-09-26 16:00:33.958369] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.965676] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:33.965685] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:33.965695] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337633965674) [2024-09-26 16:00:33.965700] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337633865461, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:33.965715] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223363201681, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:33.965720] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:33.965727] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:33.965733] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:33.965735] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:33.965741] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.965744] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:33.965749] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337633965703) [2024-09-26 16:00:33.970051] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337633801189]}) [2024-09-26 16:00:33.970676] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:33.976986] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:33.977142] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=62, refresh_score_time=24, wash_time=3) [2024-09-26 16:00:33.978061] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=125828, clean_num=31457) [2024-09-26 16:00:33.978230] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:520) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] [CACHE] tenant_id= 1 cache memory info: [CACHE] tenant_id= 1 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 27 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 8 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 393 | hold_size= 0 [2024-09-26 16:00:33.978241] INFO print_current_status (ob_kvcache_hazard_version.cpp:431) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] [KVCACHE-HAZARD] hazard version status info: current version: 16459 | min_version= 16459 | total thread store count: 62 | total nodes count: 1 | [KVCACHE-HAZARD] i= 56 | thread_id= 2513 | inited= 1 | waiting_nodes_count= 1 | last_retire_version= 16458 | acquired_version=18446744073709551615 | [2024-09-26 16:00:33.988517] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:33.988526] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:33.993032] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337633481828, now=[mts=1727337632992475]) [2024-09-26 16:00:33.993045] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=12][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337633481828, now=[mts=1727337632992475], now0=[mts=1727337632992475], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:33.993053] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337632992407, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:33.993070] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}) [2024-09-26 16:00:33.993078] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337632992407, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:33.993086] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c5654378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92c18fa400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337632992407, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337603551832, use_das=false, nested_level=0, session={this:0x7f92b33ae038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92c18fa400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:33.993102] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:33.993107] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:33.993110] WDIAG [SQL] open (ob_result_set.cpp:157) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:33.993113] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:33.993115] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:33.993119] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=29, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:33.993123] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:29, local_retry_times:29, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:33.993130] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:33.993132] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:33.993133] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:33.993140] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:33.993143] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, process_time=30441311) [2024-09-26 16:00:33.993158] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=15][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:33.993161] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name) [2024-09-26 16:00:33.993164] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:33.993166] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:33.993169] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8be0, start=1727337603551827, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name) [2024-09-26 16:00:33.993172] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:00:33.993175] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name) [2024-09-26 16:00:33.993203] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:33.993206] WDIAG [SHARE] load_refresh_schema_status (ob_schema_status_proxy.cpp:240) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4012] fail to load core table(ret=-4012) [2024-09-26 16:00:33.993208] INFO [SHARE] load_refresh_schema_status (ob_schema_status_proxy.cpp:284) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=2] [SCHEMA_STATUS] load refreshed schema status(ret=-4012) [2024-09-26 16:00:33.993216] WARN update_fail_count (ob_root_service.cpp:10374) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=2][errcode=-4752] rootservice start process has failure(msg="rootservice start()/do_restart() has failure", ret=-4012, ret="OB_TIMEOUT", fail_cnt=51) [2024-09-26 16:00:33.993254] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D6E-0-0] [lt=1][errcode=-4012] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-4012) [2024-09-26 16:00:33.993262] INFO [SHARE] store (ob_rs_mgr.cpp:128) [2579][RSAsyncTask0][T0][Y0-0006230057584E54-0-0] [lt=7] store rs list succeed(agent=0, addr_list=[{server:"192.168.0.45:2882", role:1, sql_port:2881}], force=true) [2024-09-26 16:00:33.998641] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500709, support_ssl=false) [2024-09-26 16:00:33.998678] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221500709, sess.client_addr_="127.0.0.1:36328") [2024-09-26 16:00:33.998685] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500709, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:33.998763] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:33.998772] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2688][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:33.998812] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9C-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:33.998982] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9C-0-0] [lt=3][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:33.998991] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9C-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"osrbtA%4w|*WX4:eq&hZ"}, ret=-4043) [2024-09-26 16:00:33.998997] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9C-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:33.999001] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9C-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:33.999008] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9C-0-0] [lt=1] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:33.999032] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9C-0-0] [lt=6] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500709, proxy_sessid:0}) [2024-09-26 16:00:33.999036] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9C-0-0] [lt=4] mark session id unused(sessid=3221500709) [2024-09-26 16:00:33.999040] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9C-0-0] [lt=2][errcode=0] server close connection(sessid=3221500709, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:33.999044] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9C-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500709, proxy_sessid=0) [2024-09-26 16:00:33.999046] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9C-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:33.999049] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9C-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500709, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:33.999074] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500709, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337633998727, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:33.999082] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500709, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:33.999085] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500709, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337633998727, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:33.999090] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500709, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:34.015973] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 12104.00) [2024-09-26 16:00:34.015982] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:34.015994] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:34.015998] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:34.018878] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:34.018886] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.027874] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FC-0-0] [lt=40][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:34.027883] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FC-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:34.027892] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FC-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:34.027897] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FC-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:34.036778] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:34.036791] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:34.036795] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:34.036799] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:34.036802] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:34.036804] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:34.036807] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:34.036825] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=17][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:34.036827] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:34.036829] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:34.036831] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:34.036834] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:34.036836] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:34.036838] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:34.036845] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:34.036848] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:34.036866] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=17][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:34.036868] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:34.036871] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:34.036878] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=6][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:34.036881] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:34.036886] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:34.036893] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:34.036895] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:34.036896] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:34.036904] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:34.036918] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587B-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:34.036921] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:34.036925] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587B-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:34.036929] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587B-0-0] [lt=4][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:34.036932] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587B-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:34.036935] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337634036705, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:34.036938] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587B-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:34.036941] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587B-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:34.036977] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587B-0-0] [lt=12][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:34.036981] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587B-0-0] [lt=4][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:34.047228] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:34.047525] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:34.047532] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:34.047536] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:34.047539] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:34.047542] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:34.047544] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:34.047547] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:34.047549] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:34.047551] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:34.047552] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:34.047554] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:34.047558] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:34.047560] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:34.047562] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:34.047575] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5C-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:34.049175] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:34.049184] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.065815] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:34.065831] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337634065811) [2024-09-26 16:00:34.065838] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337633965703, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:34.065880] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223363301270, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:34.065888] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:34.065898] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:34.065905] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:34.065909] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:34.065932] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.065935] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.065938] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337634065842) [2024-09-26 16:00:34.070353] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337633801189]}) [2024-09-26 16:00:34.070371] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D853-0-0] [lt=6][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337634070347}) [2024-09-26 16:00:34.079893] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:34.079901] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.081417] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:34.110014] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:34.110023] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.140066] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:34.140074] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.142927] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:34.151705] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=2] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-09-26 16:00:34.152034] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD2-0-0] [lt=8] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-09-26 16:00:34.152046] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD2-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:34.152109] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD2-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:34.166375] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:34.166388] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337634166372) [2024-09-26 16:00:34.166394] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337634065842, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:34.166420] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223363401932, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:34.166427] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:34.166446] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:34.166466] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:34.166470] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:34.166476] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.166479] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.166481] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337634166397) [2024-09-26 16:00:34.170268] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:34.170276] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.170963] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337633801189]}) [2024-09-26 16:00:34.178426] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:34.178533] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=51, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:34.179181] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=157285, clean_num=31457) [2024-09-26 16:00:34.197675] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:00:34.197685] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=8] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:34.197693] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:34.201532] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:34.201540] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.231814] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:34.231821] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.235029] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588A-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:34.235052] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588A-0-0] [lt=22][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:34.235080] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588A-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:34.262084] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:34.262092] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.267005] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:34.267017] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337634267002) [2024-09-26 16:00:34.267023] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337634166397, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:34.267039] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223363502460, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:34.267045] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:34.267052] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:34.267058] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:34.267061] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:34.267067] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.267069] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.267072] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337634267026) [2024-09-26 16:00:34.271605] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337633801189]}) [2024-09-26 16:00:34.292757] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:34.292763] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.295890] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337634224325, now=[mts=1727337633295291]) [2024-09-26 16:00:34.295900] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=9][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337634224325, now=[mts=1727337633295291], now0=[mts=1727337633295291], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:34.295906] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269011240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337633294451, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:34.295922] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269011240}) [2024-09-26 16:00:34.295930] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269011240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337633294451, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:34.295937] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269011240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337633294451, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337632294578, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269011240}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:34.295955] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:34.295960] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:34.295963] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:34.295966] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:34.295969] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:34.295973] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:34.295977] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:34.295984] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:34.295986] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:34.295988] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:34.295992] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:34.295996] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=3] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2001418) [2024-09-26 16:00:34.295999] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:34.296004] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=4][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:34.296007] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:34.296009] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:34.296012] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:34.296016] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:34.296040] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:34.296043] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:34.296046] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:34.296049] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:34.296053] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:34.296055] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:34.296176] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CDF-0-0] [lt=3][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:34.296186] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:34.296189] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:34.296214] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:34.296225] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=9][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:34.296481] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:34.296491] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=9][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:34.302248] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.302255] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337634302246], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:34.302261] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337634302246], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:34.308980] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1727337634308979, bytes: 14609914, bw: 0.003353 MB/s, add_ts: 1006997, add_bytes: 3540 [2024-09-26 16:00:34.310957] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D384C-0-0] [lt=9][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:34.323620] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:34.323633] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.343169] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:34.354962] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:00:34.354971] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.367604] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:34.367616] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337634367601) [2024-09-26 16:00:34.367622] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337634267026, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:34.367628] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:34.367638] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:34.367641] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:34.367958] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885743-0-0] [lt=5] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:34.367970] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885743-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:34.367998] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885743-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:34.368030] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337634368029, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=396, wlock_time=13, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:34.368039] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:34.368055] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223363604503, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:34.368060] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:34.368067] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:34.368073] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:34.368075] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:34.368081] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.368083] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.368085] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337634368043) [2024-09-26 16:00:34.372496] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337634302246]}) [2024-09-26 16:00:34.379435] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:34.379539] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=50, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:34.380141] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=188742, clean_num=31457) [2024-09-26 16:00:34.383228] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=5] current monitor number(seq_=-1) [2024-09-26 16:00:34.383593] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4C-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:34.383606] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4C-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:34.383611] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4C-0-0] [lt=4][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:34.383617] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4C-0-0] [lt=1] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:34.383637] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4C-0-0] [lt=0] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:34.384160] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=4] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:34.385250] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:34.385258] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.415480] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:34.415494] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.445700] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:34.445711] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.449631] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.449644] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.449650] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.450622] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.450630] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.450635] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.451639] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.451646] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.451650] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.452552] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.452557] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.452560] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.454511] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.454514] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.454516] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.455490] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.455499] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.455504] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.456460] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.456463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.456466] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.457432] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.457438] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.457441] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.458432] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.458436] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.458437] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.459400] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.459408] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.459411] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.460364] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.460369] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.460371] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.461358] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.461384] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=25][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.461386] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.462338] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.462342] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.462344] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.463281] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.463286] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.463288] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.464258] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.464263] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.464265] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.465259] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.465267] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.465280] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=13][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:34.466210] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.467196] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.467673] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:34.467688] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337634467670) [2024-09-26 16:00:34.467693] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337634367625, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:34.467718] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.467811] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223363704078, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:34.467819] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:34.467828] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:34.467833] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:34.467836] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:34.467842] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.467845] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.467849] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337634467697) [2024-09-26 16:00:34.468667] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.470102] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.471089] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.473090] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.473117] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337634302246]}) [2024-09-26 16:00:34.474014] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.475014] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.475987] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.475987] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:34.475993] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.477802] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.478927] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.479882] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.480860] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.481841] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.482800] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.483362] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.483961] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.484546] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.485750] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.486720] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.487718] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.488692] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.489680] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.491622] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.492583] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.493663] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.494457] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.495523] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.496472] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.497464] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.498480] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.499495] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.500090] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.500790] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.501473] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.502403] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.504352] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.505306] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.506269] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.506314] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:34.506321] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.508196] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.509155] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.511100] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.512127] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.513098] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.514079] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.515106] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.516046] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.518960] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.519949] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.520928] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.522875] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.523842] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.524821] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.525827] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=36][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.525958] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FD-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:34.525966] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FD-0-0] [lt=7][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:34.525975] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FD-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:34.525983] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FD-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:34.526769] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.527754] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.528746] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.529701] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.530708] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.531666] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.532639] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.533613] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.535588] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.536617] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.536621] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:34.536628] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.537528] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.539513] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.540456] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.540511] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6369, mem_used=12478464) [2024-09-26 16:00:34.541427] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.542469] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=34][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.543353] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:34.543374] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.544359] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.545341] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.546341] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.547316] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.548286] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.549234] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.551238] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.552165] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.553224] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.555094] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.556098] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.557174] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.558019] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:34.561969] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:00:34.566815] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:34.566831] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.567856] WDIAG [PALF] convert_to_ts (scn.cpp:265) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2024-09-26 16:00:34.567862] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1727337634567854, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1727337634567854, force_self_check=false, weak_read_refresh_interval=100000) [2024-09-26 16:00:34.567876] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:34.567882] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337634567874) [2024-09-26 16:00:34.567885] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337634467697, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:34.567901] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223363804326, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:34.567906] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:34.567914] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:34.567918] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:34.567921] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:34.567926] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.567929] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.567930] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337634567888) [2024-09-26 16:00:34.574674] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337634302246]}) [2024-09-26 16:00:34.580614] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:34.580718] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=3) [2024-09-26 16:00:34.581320] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=220199, clean_num=31457) [2024-09-26 16:00:34.597088] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:34.597097] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.602000] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.602003] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.602011] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.602039] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.627442] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:34.627461] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.649876] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:34.655732] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:34.658647] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:34.658655] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.668426] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:34.668445] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337634668422) [2024-09-26 16:00:34.668450] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337634567887, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:34.668449] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:34.668473] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get_max_applied_scn(scn={val:1727114270763524000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:374, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:1727114270763524000, v:0}, max_applied_cb_scn_:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:34.668483] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get_min_unreplayed_log_info(lsn={lsn:629872191939}, scn={val:1727114270763524001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:00:34.668503] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=223363904948, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:34.668509] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:34.668516] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:34.668521] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:34.668524] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:34.668529] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.668532] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.668534] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337634668453) [2024-09-26 16:00:34.675240] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337634302246]}) [2024-09-26 16:00:34.688890] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:34.688898] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.706515] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:155, tid:2736}]) [2024-09-26 16:00:34.719220] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:34.719232] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.739712] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337634739711, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007960, add_bytes: 0 [2024-09-26 16:00:34.743618] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:34.749468] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:34.749477] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.757283] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=2] XA scheduler heartbeat task statistics(avg_time=1) [2024-09-26 16:00:34.769002] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:34.769013] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337634768999) [2024-09-26 16:00:34.769019] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337634668453, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:34.769036] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223364004499, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:34.769043] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:34.769051] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:34.769057] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:34.769060] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:34.769066] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.769068] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.769070] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337634769022) [2024-09-26 16:00:34.775837] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337634302246]}) [2024-09-26 16:00:34.779723] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=4] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-09-26 16:00:34.779734] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-09-26 16:00:34.779740] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=4] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=687) [2024-09-26 16:00:34.779761] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:34.779765] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.781810] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:34.781913] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=55, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:34.782528] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=251656, clean_num=31457) [2024-09-26 16:00:34.803310] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:34.803320] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337634803308], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:34.803329] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337634803308], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:34.803332] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:34.803336] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:34.810050] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:34.810067] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.821739] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.821759] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.821830] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.821843] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.821851] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.821857] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.821862] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.821867] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.821872] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.821876] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.821907] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:34.841309] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:34.841316] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.846157] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=21] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:34.863087] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, table_name.ptr()="data_size:16, data:5F5F616C6C5F6D657267655F696E666F", ret=-5019) [2024-09-26 16:00:34.863102] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, ret=-5019) [2024-09-26 16:00:34.863106] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_merge_info, db_name=oceanbase) [2024-09-26 16:00:34.863122] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_merge_info) [2024-09-26 16:00:34.863126] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:34.863128] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:34.863131] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_merge_info' doesn't exist [2024-09-26 16:00:34.863134] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:34.863136] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:34.863138] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:34.863140] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:34.863142] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:34.863145] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:34.863147] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:34.863154] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:34.863157] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:34.863161] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:34.863164] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:34.863167] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_merge_info WHERE tenant_id = '1', ret=-5019) [2024-09-26 16:00:34.863169] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:34.863172] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:34.863177] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:34.863187] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:34.863190] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:34.863192] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:34.863197] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:34.863201] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=4][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:34.863204] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885080-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, aret=-5019, ret=-5019) [2024-09-26 16:00:34.863207] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:34.863210] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:34.863212] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:34.863215] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337634863018, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:34.863218] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:34.863221] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:49) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:34.863263] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:890) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] failed to load global merge info(ret=-5019, ret="OB_TABLE_NOT_EXIST", global_merge_info={tenant_id:1, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2024-09-26 16:00:34.863288] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1005) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-5019] fail to refresh merge info(tmp_ret=-5019, tmp_ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:34.863520] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:34.863527] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:34.871546] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:34.871556] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337634871543) [2024-09-26 16:00:34.871585] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337634769022, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:34.871590] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:34.871604] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:34.871607] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:34.871606] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:34.871612] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.871846] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:34.871855] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:34.871858] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:34.871861] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:34.871863] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:34.871865] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:34.871868] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:34.871870] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:34.871872] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:34.871874] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:34.871876] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:34.871878] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:34.871880] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:34.871882] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:34.871886] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:34.871888] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:34.871890] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:34.871893] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:34.871895] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:34.871897] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:34.871900] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:34.871904] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:34.871907] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:34.871909] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:34.871911] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:34.871916] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:34.871919] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:34.871921] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885744-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:34.871926] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:34.871929] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:34.871931] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:34.871933] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337634871798, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:34.871935] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:34.871938] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:34.871957] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:34.871969] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337634871968, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=370, wlock_time=11, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:34.871975] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:34.871978] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:34.871992] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223364108028, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:34.871997] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:34.872003] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:34.872007] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:34.872010] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:34.872015] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.872017] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.872019] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337634871982) [2024-09-26 16:00:34.876511] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337634803308]}) [2024-09-26 16:00:34.901824] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:34.901833] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.916474] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] ====== checkpoint timer task ====== [2024-09-26 16:00:34.916888] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] succ to get rec scn(*this={this:0x7f92a6a04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1726077600526835000, v:0}, last_push_gc_task_ts_:1727334198745636, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:34.916901] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13] tx ctx memtable get rec scn(this={ObITable:{this:0x7f92a22a4080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f92a22a4080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:34.916914] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1723174350926847000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-09-26 16:00:34.916930] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:34.916952] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}, min_rec_scn={val:1726163586617842000, v:0}, ls_id_={id:1}) [2024-09-26 16:00:34.916965] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:34.916968] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:34.916971] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:34.916985] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1726163586617842000, v:0}, checkpoint_scn_in_ls_meta={val:1726163586617842000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-09-26 16:00:34.916994] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}, curr_base_lsn={lsn:627966418944}, new_base_lsn={lsn:627966418944}, lsn={lsn:627966418944}) [2024-09-26 16:00:34.916999] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:627966418944}, ls->get_ls_id()={id:1}) [2024-09-26 16:00:34.917003] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-09-26 16:00:34.932131] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:34.932137] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.932155] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:34.943818] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:34.962382] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:34.962389] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.972176] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:34.972183] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:34.972184] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:34.972197] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337634972174) [2024-09-26 16:00:34.972202] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337634871588, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:34.972217] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223364207721, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:34.972223] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:34.972230] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:34.972235] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:34.972237] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:34.972242] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.972245] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:34.972247] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337634972205) [2024-09-26 16:00:34.977046] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337634803308]}) [2024-09-26 16:00:34.978130] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=3] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-09-26 16:00:34.978139] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=7] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:34.978144] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=4] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:34.978148] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=3] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:34.979786] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] Dumping All Cache Objs(alloc_obj_list.count()=3, alloc_obj_list=[{obj_id:78, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157779}, {obj_id:79, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:4, added_to_lc:true, mem_used:175172}]) [2024-09-26 16:00:34.979801] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=13] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:34.983080] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:34.983182] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:34.983825] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=283113, clean_num=31457) [2024-09-26 16:00:34.992667] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:34.992674] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:34.999851] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500716, support_ssl=false) [2024-09-26 16:00:34.999887] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221500716, sess.client_addr_="127.0.0.1:36330") [2024-09-26 16:00:34.999893] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f9264404030, session_id:3221500716, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:34.999912] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:34.999922] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:34.999925] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:280) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=3] mysql login queue(mysql_queue->queue_.size()=1) [2024-09-26 16:00:35.000042] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9D-0-0] [lt=8] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:35.000268] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9D-0-0] [lt=7][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:35.000275] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9D-0-0] [lt=6][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"DbCfWclt!Dt:elpmT;j'"}, ret=-4043) [2024-09-26 16:00:35.000281] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9D-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:35.000285] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9D-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:35.000323] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9D-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:35.000347] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9D-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500716, proxy_sessid:0}) [2024-09-26 16:00:35.000351] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9D-0-0] [lt=3] mark session id unused(sessid=3221500716) [2024-09-26 16:00:35.000354] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9D-0-0] [lt=2][errcode=0] server close connection(sessid=3221500716, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:35.000358] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9D-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500716, proxy_sessid=0) [2024-09-26 16:00:35.000360] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9D-0-0] [lt=1][errcode=-4016] session is null [2024-09-26 16:00:35.000362] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9D-0-0] [lt=1] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500716, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:35.000381] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f9264404030, session_id:3221500716, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337634999125, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:35.000388] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500716, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:35.000391] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9264404030, session_id:3221500716, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337634999125, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:35.000403] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500716, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:35.022327] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:35.022343] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=14][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-09-26 16:00:35.022348] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:35.022352] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-09-26 16:00:35.022368] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=14][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:35.022370] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:35.022374] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-09-26 16:00:35.022376] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:35.022378] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:35.022380] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:35.022382] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] resolve table failed(ret=-5019) [2024-09-26 16:00:35.022444] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:2822) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=61][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-09-26 16:00:35.022446] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2676) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] resolve joined table failed(ret=-5019) [2024-09-26 16:00:35.022448] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:35.022450] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:35.022452] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:35.022454] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:35.022462] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:35.022464] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:35.022467] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:35.022469] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:35.022496] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=26][errcode=-5019] fail to handle text query(stmt=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-09-26 16:00:35.022499] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:35.022502] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:35.022509] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:35.022517] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:35.022519] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:35.022521] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:35.022528] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:35.022532] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9D-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:35.022536] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [3024][BlackListServic][T0][YB42C0A8002D-0006230056284F9D-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-09-26 16:00:35.022539] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:35.022542] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:35.022538] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 9333.00) [2024-09-26 16:00:35.022544] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:35.022547] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337635022210, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:35.022551] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:35.022553] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:222) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:35.022546] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:35.022559] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:35.022564] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:35.022581] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3] ls blacklist refresh finish(cost_time=614) [2024-09-26 16:00:35.022906] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:35.022926] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.025083] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FE-0-0] [lt=37][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:35.025092] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FE-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:35.025110] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FE-0-0] [lt=17][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:35.025114] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FE-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:35.037892] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:35.037904] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:35.037908] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:35.037911] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:35.037915] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:35.037917] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:35.037920] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:35.037923] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:35.037925] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:35.037927] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:35.037929] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:35.037931] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:35.037933] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:35.037936] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:35.037941] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:35.037944] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:35.037946] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:35.037951] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:35.037954] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:35.037956] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:35.037958] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:35.037964] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:35.037969] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:35.037971] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:35.037973] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:35.037978] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:35.037981] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587C-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:35.037985] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587C-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:35.037987] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:35.037990] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587C-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:35.038004] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587C-0-0] [lt=14][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:35.038007] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587C-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337635037834, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:35.038011] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587C-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:35.038015] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587C-0-0] [lt=4][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:35.038037] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587C-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:35.038039] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587C-0-0] [lt=2][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:35.048354] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=17] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:35.048669] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=13] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:35.048679] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:35.048683] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:35.048686] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:35.048688] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:35.048690] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:35.048693] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:35.048695] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:35.048697] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:35.048699] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:35.048701] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:35.048703] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:35.048705] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:35.048707] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:35.048712] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:35.048714] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:35.048719] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:35.048722] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:35.048724] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:35.048726] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:35.048728] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:35.048733] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:35.048738] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:35.048740] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:35.048741] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:35.048746] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:35.048750] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E5E-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:35.048753] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2582][RSAsyncTask3][T0][YB42C0A8002D-0006230057684E5E-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:35.048756] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:35.048758] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:35.048763] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:35.048766] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337635048628, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:35.048769] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:35.048813] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:35.053209] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:35.053218] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.072773] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:35.072783] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337635072770) [2024-09-26 16:00:35.072789] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337634972205, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:35.072806] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223364309250, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.072813] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:35.072822] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:35.072827] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:35.072830] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:35.072837] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.072839] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.072841] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337635072791) [2024-09-26 16:00:35.077644] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337634803308]}) [2024-09-26 16:00:35.077664] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D85F-0-0] [lt=6][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337635077637}) [2024-09-26 16:00:35.081569] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=5] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:35.083525] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:35.083531] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.113813] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:35.113821] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.140151] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:221) [2596][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=11] [ASYNC TASK QUEUE](queue_size=0) [2024-09-26 16:00:35.144041] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:35.144051] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.145038] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=15] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:35.173364] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:35.173388] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337635173350) [2024-09-26 16:00:35.173394] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337635072791, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:35.173421] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223364408941, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.173429] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:35.173438] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:35.173444] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:35.173447] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:35.173454] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.173456] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.173458] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337635173398) [2024-09-26 16:00:35.174327] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:35.174334] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.178300] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337634803308]}) [2024-09-26 16:00:35.184215] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:35.184321] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:35.184923] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=314570, clean_num=31457) [2024-09-26 16:00:35.201928] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:35.201940] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:35.201944] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:35.204606] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:35.204613] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.234907] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:35.234915] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.235680] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:35.235692] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:35.235696] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:35.235699] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:35.235703] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:35.235705] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:35.235709] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:35.235711] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:35.235713] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:35.235715] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:35.235717] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:35.235736] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=18][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.235759] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588B-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:35.265173] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:35.265199] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.273476] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:35.273489] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337635273473) [2024-09-26 16:00:35.273494] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337635173397, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:35.273512] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223364509438, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.273518] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:35.273525] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:35.273530] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:35.273533] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:35.273538] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.273541] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.273543] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337635273497) [2024-09-26 16:00:35.278508] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337634803308]}) [2024-09-26 16:00:35.295340] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:35.295348] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.297386] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337636226233, now=[mts=1727337634296479]) [2024-09-26 16:00:35.297394] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337636226233, now=[mts=1727337634296479], now0=[mts=1727337634296479], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:35.297401] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92691916e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337634295877, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:35.297440] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=37][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92691916e0}) [2024-09-26 16:00:35.297448] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92691916e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337634295877, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:35.297456] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92691916e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337634295877, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337634296454, use_das=false, nested_level=0, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92691916e0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:35.297473] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:35.297477] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:35.297480] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:35.297483] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:35.297486] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:35.297490] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:35.297495] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=998739, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337636296233) [2024-09-26 16:00:35.297499] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:35.297506] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:35.297508] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:35.297509] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:35.297515] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:35.297551] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:35.297554] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:35.304384] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.304410] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337635304381], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:35.304417] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337635304381], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:35.304424] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337635304381]}) [2024-09-26 16:00:35.309129] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=13] PNIO [ratelimit] time: 1727337635309128, bytes: 14613736, bw: 0.003644 MB/s, add_ts: 1000149, add_bytes: 3822 [2024-09-26 16:00:35.312333] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D38C2-0-0] [lt=14][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:35.327687] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:35.327698] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.345139] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:35.357984] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:00:35.358009] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.373637] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:35.373651] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337635373633) [2024-09-26 16:00:35.373657] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337635273497, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:35.373664] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:35.373674] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:35.373677] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:35.373996] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885745-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:35.374006] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885745-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.374035] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885745-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:35.374071] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337635374070, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=401, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:35.374080] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:35.374096] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223364610045, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.374102] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:35.374109] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:35.374114] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:35.374116] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:35.374122] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.374125] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.374127] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337635374085) [2024-09-26 16:00:35.380465] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337635304381]}) [2024-09-26 16:00:35.384282] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=10] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:35.385446] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:35.385589] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=54, refresh_score_time=34, wash_time=2) [2024-09-26 16:00:35.386221] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=346027, clean_num=31457) [2024-09-26 16:00:35.388231] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:35.388236] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.418326] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:35.418335] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.448791] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:35.448802] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.450752] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.450761] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.450768] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.451302] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.451305] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.451307] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.451315] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.451316] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.451318] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.452680] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.452685] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.452687] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.452691] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.452703] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.452705] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.453627] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.453632] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.453634] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.453637] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.453639] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.453640] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.454656] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.454665] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.454670] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.454674] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.454677] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.454679] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.455666] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.455674] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.455677] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.455681] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.455701] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.455705] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.456583] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.456588] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.456590] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.456594] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.456595] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.456597] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.457564] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.457569] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.457571] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.457575] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.457576] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.457578] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.458587] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.458593] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.458596] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:35.458600] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.460517] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.460522] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.461504] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] ====== tenant freeze timer task ====== [2024-09-26 16:00:35.461509] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.461514] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:35.461517] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.461543] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337630620744, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337630620709, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337630620761, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:35.461568] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:35.461631] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D6-0-0] [lt=4] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:35.461642] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D6-0-0] [lt=8] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:35.461648] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D6-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:35.461651] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D6-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:35.461654] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D6-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:35.461657] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D6-0-0] [lt=3][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:35.461659] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D6-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:35.461669] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D6-0-0] [lt=9] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:35.461671] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D6-0-0] [lt=1][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:35.461673] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D6-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:35.461676] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D6-0-0] [lt=1][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:35.461712] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851D6-0-0] [lt=3] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:35.462459] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.462464] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.463429] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.463437] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.464402] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.464409] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.465175] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.465182] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.466485] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.466495] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.467217] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.467226] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.468331] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.468337] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.469291] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.469297] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.470269] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.470274] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.471234] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.471240] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.472215] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [2925][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=19] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f92b77d02b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-09-26 16:00:35.473195] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.473212] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.474231] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:35.474236] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.474240] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.474245] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337635474228) [2024-09-26 16:00:35.474250] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337635373661, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:35.474269] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223364710710, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.474275] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:35.474292] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:35.474298] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:35.474301] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:35.474307] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.474309] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.474311] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337635474254) [2024-09-26 16:00:35.475136] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.475142] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.476205] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.476210] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.476776] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.476784] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.478076] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.478080] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.479088] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:35.479095] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.479106] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.479113] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.480029] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.480036] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.480585] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.480596] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337635304381]}) [2024-09-26 16:00:35.480608] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.482012] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.482020] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.482966] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.482976] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.483546] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.483553] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.484976] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.484982] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.486985] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.486992] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.487836] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.487844] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.488838] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.488847] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.489796] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.489803] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.490316] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.490323] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.491753] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.491762] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.492727] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.492735] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.493751] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.493759] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.494464] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.494471] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.495175] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.495182] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.496635] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.496644] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.497261] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.497268] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.498601] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.498605] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.499561] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.499567] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.501548] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.501553] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.502514] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.502531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.503267] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:35.510343] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:35.510349] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.528763] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FF-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:35.528774] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FF-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:35.528782] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FF-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:35.528787] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-00062300571858FF-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:35.540595] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6377, mem_used=12478464) [2024-09-26 16:00:35.541578] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16] success to notify_need_writing_throttling True [2024-09-26 16:00:35.541585] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.545717] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=692032, replace_num=15728) [2024-09-26 16:00:35.545743] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:35.562086] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=8] gc stale ls task succ [2024-09-26 16:00:35.571856] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:35.571863] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.574794] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:35.574815] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337635574790) [2024-09-26 16:00:35.574823] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337635474254, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:35.574850] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223364811280, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.574857] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:35.574865] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:35.574873] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:35.574876] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:35.574882] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.574886] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.574889] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337635574826) [2024-09-26 16:00:35.580678] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337635304381]}) [2024-09-26 16:00:35.586615] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:35.586749] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=53, refresh_score_time=26, wash_time=1) [2024-09-26 16:00:35.587363] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=377484, clean_num=31457) [2024-09-26 16:00:35.589471] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [2619][DumpLSLoc][T0][YB42C0A8002D-0006230058384C79-0-0] [lt=13] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1727337630931506, replica_locations:[{server:"192.168.0.45:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:374}]}]) [2024-09-26 16:00:35.598227] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [2635][LocalityReload][T0][Y0-0000000000000000-0-0] [lt=7] runTimer to refresh locality_info(ret=0) [2024-09-26 16:00:35.598622] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C79-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:35.598631] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C79-0-0] [lt=7][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.598659] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C79-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:35.598666] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=0] load region(ret=-5019, locality_info={version:0, local_region:"", local_zone:"", local_idc:"", local_zone_type:3, local_zone_status:3, locality_region_array:[], locality_zone_array:[]}) [2024-09-26 16:00:35.602132] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.602136] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.602147] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.602154] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:35.602154] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.602160] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.622477] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] ====== traversal_flush timer task ====== [2024-09-26 16:00:35.622503] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] add dag success(dag=0x7f9264c3a030, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6163825430799493735, dag_cnt=1, dag_type_cnts=1) [2024-09-26 16:00:35.622510] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] success to create and add dag(ret=0, dag=0x7f9264c3a030) [2024-09-26 16:00:35.622515] INFO [STORAGE] flush (ob_tx_data_memtable.cpp:917) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] [TX DATA MERGE]schedule flush tx data memtable task done(ret=0, ret="OB_SUCCESS", get_ls_id()={id:1}, this=0x7f92a2294080, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:4, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337635622515, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:35.622550] INFO [MDS] flush (mds_table_mgr.cpp:172) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=19] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:35.622565] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337635622560, task_id:YB42C0A8002D-0006230056D8C616-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.622607] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=34] schedule one task(task=0x7f9264c3a3f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.622643] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=49] add dag success(dag=0x7f9264c3a560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6659188001345271846, dag_cnt=2, dag_type_cnts=1) [2024-09-26 16:00:35.622648] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] success to create and add dag(ret=0, dag=0x7f9264c3a560) [2024-09-26 16:00:35.622651] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, create_flush_dag_time:1727337635622651, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452867, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200240}, checkpoint_unit_ptr:0x7f92a1a70850}, is_freeze_clock:true}) [2024-09-26 16:00:35.622658] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337635622656, task_id:YB42C0A8002D-0006230056D8C617-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.622670] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=12] schedule one task(task=0x7f9264c3a920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.622661] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=10] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337635622633, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.622702] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=15] add dag success(dag=0x7f9264c3aa90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-9069613569758001279, dag_cnt=3, dag_type_cnts=2) [2024-09-26 16:00:35.622708] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] success to create and add dag(ret=0, dag=0x7f9264c3aa90) [2024-09-26 16:00:35.622711] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, create_flush_dag_time:1727337635622710, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452892, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200489}, checkpoint_unit_ptr:0x7f92a1a70080}, is_freeze_clock:true}) [2024-09-26 16:00:35.622714] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337635622676, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.622721] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337635622720, task_id:YB42C0A8002D-0006230056D8C618-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.622731] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=10] schedule one task(task=0x7f9264c3ae50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.622766] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=14][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.622772] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.622775] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:35.622785] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.622983] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.622993] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=10][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.622997] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726157244792838000) [2024-09-26 16:00:35.622997] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=10] add dag success(dag=0x7f9264c3afc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1288986994123435108, dag_cnt=4, dag_type_cnts=3) [2024-09-26 16:00:35.623000] INFO [STORAGE] ready_for_flush (ob_tx_data_memtable.cpp:865) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=3] memtable is frozen yet.(this=0x7f92a2294080) [2024-09-26 16:00:35.623002] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] success to create and add dag(ret=0, dag=0x7f9264c3afc0) [2024-09-26 16:00:35.623031] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.623034] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.623035] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:35.623039] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.623006] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337635623006, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452911, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847222}, checkpoint_unit_ptr:0x7f92a1a71020}, is_freeze_clock:true}) [2024-09-26 16:00:35.623046] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=39] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337635622724, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623063] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=12] add dag success(dag=0x7f9264c3b4f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5355160120942094702, dag_cnt=5, dag_type_cnts=4) [2024-09-26 16:00:35.623066] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3b4f0) [2024-09-26 16:00:35.623067] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337635623067, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452929, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847223}, checkpoint_unit_ptr:0x7f92a1a717f0}, is_freeze_clock:true}) [2024-09-26 16:00:35.623071] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337635623058, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623081] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405042) [2024-09-26 16:00:35.623123] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:35.623115] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337635623111, task_id:YB42C0A8002D-0006230056D8C619-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.623133] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=17] schedule one task(task=0x7f9264c3b380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.623140] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337635623139, task_id:YB42C0A8002D-0006230056D8C61A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.623143] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=3] schedule one task(task=0x7f9264c3b8b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.623169] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.623169] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.623173] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.623173] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.623175] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:35.623176] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:35.623179] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.623181] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.623185] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847222}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:35.623196] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.623184] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726164780922097000, old_tablet_snaphsot_version=1726164781026522000, tablet={this:0x7f92b863df20, is_inited:true, wash_score:1727337635621733000, hold_ref_cnt:true, ref_cnt:2, version:2, length:487, tablet_addr:[816017-1849-0-2305843626616512512-2](offset=1269760,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, data_tablet_id:{id:200240}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726164781225658000, v:0}, clog_checkpoint_scn:{val:1726164781225658000, v:0}, ddl_checkpoint_scn:{val:1711609700160391001, v:0}, snapshot_version:1726164781026522000, multi_version_start:1726164780922097000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726164780922097000, cur_report_version:1726164780922097000, data_checksum:2501178096, row_count:147053}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:1711609700101236001, v:0}, ddl_snapshot_version:1711609700074566000, max_sync_storage_schema_version:1711609699705840, max_serialized_medium_scn:0, ddl_execution_id:0, ddl_data_format_version:17180000769, ddl_commit_scn:{val:1711609700160391001, v:0}, mds_checkpoint_scn:{val:1726201924319685000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1711609699705840}, table_store_addr:{ptr:{this:0x7f92b863e5d8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844241870487552-0](offset=1179648,size=1148,type=2,seq=0), type:"MAJOR", tablet_id:{id:1152921504606847223}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726164780922097000, v:0}}, snapshot_version:1726164780922097000, max_merge_version:1726164780922097000}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337635623058, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843251880595456-2](offset=1249280,size=226,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276217058-0](offset=1249506,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255101821304-0](offset=1249656,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:1726164780922097000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1711609699719192000, v:0}, create_commit_version:1711609699719192000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a717f0, 0x7f92a1a72f60, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.623241] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:35.623249] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.623263] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405070) [2024-09-26 16:00:35.623272] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:35.623279] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405087) [2024-09-26 16:00:35.623285] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:35.623289] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405104) [2024-09-26 16:00:35.623295] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:35.623301] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] add dag success(dag=0x7f9264c3ba20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1994541446894784994, dag_cnt=6, dag_type_cnts=5) [2024-09-26 16:00:35.623304] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c3ba20) [2024-09-26 16:00:35.623307] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, create_flush_dag_time:1727337635623306, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209556884, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:226}, checkpoint_unit_ptr:0x7f92a1a73f00}, is_freeze_clock:true}) [2024-09-26 16:00:35.623310] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337635623297, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623327] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=10] add dag success(dag=0x7f9264c3bf50, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8791616776033068452, dag_cnt=7, dag_type_cnts=6) [2024-09-26 16:00:35.623329] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3bf50) [2024-09-26 16:00:35.623331] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337635623331, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405141, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847408}, checkpoint_unit_ptr:0x7f92a1a746d0}, is_freeze_clock:true}) [2024-09-26 16:00:35.623334] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337635623320, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623352] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635623343, task_id:YB42C0A8002D-0006230056D8C61B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.623357] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=4] schedule one task(task=0x7f9264c3bde0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.623367] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=10] add dag success(dag=0x7f9264c3c480, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6550230011160095889, dag_cnt=8, dag_type_cnts=7) [2024-09-26 16:00:35.623369] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3c480) [2024-09-26 16:00:35.623370] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337635623370, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405158, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847409}, checkpoint_unit_ptr:0x7f92a1a75670}, is_freeze_clock:true}) [2024-09-26 16:00:35.623374] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337635623344, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623386] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3c9b0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6809498982591245614, dag_cnt=9, dag_type_cnts=8) [2024-09-26 16:00:35.623388] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3c9b0) [2024-09-26 16:00:35.623390] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, create_flush_dag_time:1727337635623390, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405174, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:200004}, checkpoint_unit_ptr:0x7f92a1a74ea0}, is_freeze_clock:true}) [2024-09-26 16:00:35.623393] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337635623381, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623420] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=21] add dag success(dag=0x7f9264c3cee0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6815944285528637599, dag_cnt=10, dag_type_cnts=9) [2024-09-26 16:00:35.623425] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] success to create and add dag(ret=0, dag=0x7f9264c3cee0) [2024-09-26 16:00:35.623427] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, create_flush_dag_time:1727337635623427, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209506214, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1}, checkpoint_unit_ptr:0x7f92a1a76610}, is_freeze_clock:true}) [2024-09-26 16:00:35.623430] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337635623415, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623441] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3d410, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-4182959058374342994, dag_cnt=11, dag_type_cnts=10) [2024-09-26 16:00:35.623445] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3d410) [2024-09-26 16:00:35.623447] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, create_flush_dag_time:1727337635623447, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405204, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:412}, checkpoint_unit_ptr:0x7f92a1a75e40}, is_freeze_clock:true}) [2024-09-26 16:00:35.623450] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337635623437, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623489] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=28] add dag success(dag=0x7f9264c3d940, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1586518267935340, dag_cnt=12, dag_type_cnts=11) [2024-09-26 16:00:35.623492] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c3d940) [2024-09-26 16:00:35.623493] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, create_flush_dag_time:1727337635623493, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405220, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:444}, checkpoint_unit_ptr:0x7f92a1a775b0}, is_freeze_clock:true}) [2024-09-26 16:00:35.623496] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337635623478, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623509] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c3de70, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3044367941103776476, dag_cnt=13, dag_type_cnts=12) [2024-09-26 16:00:35.623512] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c3de70) [2024-09-26 16:00:35.623513] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, create_flush_dag_time:1727337635623513, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455964, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:309}, checkpoint_unit_ptr:0x7f92a1a76de0}, is_freeze_clock:true}) [2024-09-26 16:00:35.623517] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337635623504, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623526] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c3e3a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1596830851104282047, dag_cnt=14, dag_type_cnts=13) [2024-09-26 16:00:35.623531] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c3e3a0) [2024-09-26 16:00:35.623533] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, create_flush_dag_time:1727337635623532, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455986, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:331}, checkpoint_unit_ptr:0x7f92a1a77d80}, is_freeze_clock:true}) [2024-09-26 16:00:35.623535] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337635623523, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623545] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3e8d0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2360195962081676673, dag_cnt=15, dag_type_cnts=14) [2024-09-26 16:00:35.623559] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=11] success to create and add dag(ret=0, dag=0x7f9264c3e8d0) [2024-09-26 16:00:35.623561] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, create_flush_dag_time:1727337635623561, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405262, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:458}, checkpoint_unit_ptr:0x7f92a1a7a490}, is_freeze_clock:true}) [2024-09-26 16:00:35.623565] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337635623542, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623579] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264c3ee00, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5675615614797020460, dag_cnt=16, dag_type_cnts=15) [2024-09-26 16:00:35.623584] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3ee00) [2024-09-26 16:00:35.623586] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, create_flush_dag_time:1727337635623586, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405279, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:487}, checkpoint_unit_ptr:0x7f92a1a79cc0}, is_freeze_clock:true}) [2024-09-26 16:00:35.623589] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337635623574, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623603] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c3f330, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-612435342908088006, dag_cnt=17, dag_type_cnts=16) [2024-09-26 16:00:35.623606] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c3f330) [2024-09-26 16:00:35.623608] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, create_flush_dag_time:1727337635623608, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405296, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:455}, checkpoint_unit_ptr:0x7f92a1a794f0}, is_freeze_clock:true}) [2024-09-26 16:00:35.623611] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337635623598, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623613] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.623618] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.623621] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.623623] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3f860, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7868818205990873505, dag_cnt=18, dag_type_cnts=17) [2024-09-26 16:00:35.623626] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3f860) [2024-09-26 16:00:35.623625] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.623627] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, create_flush_dag_time:1727337635623627, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405313, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:492}, checkpoint_unit_ptr:0x7f92a1a78550}, is_freeze_clock:true}) [2024-09-26 16:00:35.623630] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337635623618, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623640] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3fd90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7071731214611448811, dag_cnt=19, dag_type_cnts=18) [2024-09-26 16:00:35.623645] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3fd90) [2024-09-26 16:00:35.623646] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, create_flush_dag_time:1727337635623646, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405347, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:501}, checkpoint_unit_ptr:0x7f92a1a78d20}, is_freeze_clock:true}) [2024-09-26 16:00:35.623649] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337635623637, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623662] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264c402c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8731750006000903454, dag_cnt=20, dag_type_cnts=19) [2024-09-26 16:00:35.623664] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c402c0) [2024-09-26 16:00:35.623666] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, create_flush_dag_time:1727337635623666, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405365, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:472}, checkpoint_unit_ptr:0x7f92a1a7ac60}, is_freeze_clock:true}) [2024-09-26 16:00:35.623629] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861df20, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:502, tablet_addr:[816017-1849-0-2305843642722988032-2](offset=1617920,size=590,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:226}, data_tablet_id:{id:226}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726178585269925002, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726178585269925002, cur_report_version:1726178585269925002, data_checksum:1151028828, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726201924319685000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861e5fc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844296631750656-0](offset=1609728,size=1199,type=2,seq=0), type:"MAJOR", tablet_id:{id:226}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726178585269925002, v:0}}, snapshot_version:1726178585269925002, max_merge_version:1726178585269925002}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337635623297, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcc550, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241777443612000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209404959}, this:0x7f92a3bcc550, timestamp:1727334209404959, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:52, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:27262976}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277811954486000, v:0}, rec_scn:{val:1726241777645865000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843250807218176-2](offset=1613824,size=225,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719032545-0](offset=1614049,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880960381-0](offset=1614205,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447480415-0](offset=1614431,size=193,type=2,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:1726178585269925002}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a73f00, 0x7f92a3bcc550, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.623669] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337635623658, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623673] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=43] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:00:35.623679] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241777443612000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.623680] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c407f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6766388540859538791, dag_cnt=21, dag_type_cnts=20) [2024-09-26 16:00:35.623682] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c407f0) [2024-09-26 16:00:35.623684] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, create_flush_dag_time:1727337635623684, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405384, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:342}, checkpoint_unit_ptr:0x7f92a1a7cba0}, is_freeze_clock:true}) [2024-09-26 16:00:35.623687] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337635623676, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623719] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=27] add dag success(dag=0x7f9264c40d20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2986199386776543176, dag_cnt=22, dag_type_cnts=21) [2024-09-26 16:00:35.623723] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c40d20) [2024-09-26 16:00:35.623725] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337635623725, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405418, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:140}, checkpoint_unit_ptr:0x7f92a1a7b430}, is_freeze_clock:true}) [2024-09-26 16:00:35.623728] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337635623714, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623740] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c41250, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6947063304478012270, dag_cnt=23, dag_type_cnts=22) [2024-09-26 16:00:35.623743] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c41250) [2024-09-26 16:00:35.623745] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337635623744, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405436, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101009}, checkpoint_unit_ptr:0x7f92a1a7bc00}, is_freeze_clock:true}) [2024-09-26 16:00:35.623748] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337635623735, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623759] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c41780, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5843378789243662202, dag_cnt=24, dag_type_cnts=23) [2024-09-26 16:00:35.623762] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c41780) [2024-09-26 16:00:35.623763] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337635623763, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405453, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101010}, checkpoint_unit_ptr:0x7f92a1a7c3d0}, is_freeze_clock:true}) [2024-09-26 16:00:35.623767] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337635623754, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623777] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c41cb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1841823684513746176, dag_cnt=25, dag_type_cnts=24) [2024-09-26 16:00:35.623781] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c41cb0) [2024-09-26 16:00:35.623783] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, create_flush_dag_time:1727337635623783, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405470, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:323}, checkpoint_unit_ptr:0x7f92a1a7d370}, is_freeze_clock:true}) [2024-09-26 16:00:35.623786] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337635623773, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623803] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c421e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-2678407261265885087, dag_cnt=26, dag_type_cnts=25) [2024-09-26 16:00:35.623805] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264c421e0) [2024-09-26 16:00:35.623806] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, create_flush_dag_time:1727337635623806, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405487, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:377}, checkpoint_unit_ptr:0x7f92a1a7e310}, is_freeze_clock:true}) [2024-09-26 16:00:35.623809] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337635623795, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623820] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c42710, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7175089551615529855, dag_cnt=27, dag_type_cnts=26) [2024-09-26 16:00:35.623821] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264c42710) [2024-09-26 16:00:35.623823] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, create_flush_dag_time:1727337635623823, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405503, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:297}, checkpoint_unit_ptr:0x7f92a1a7eae0}, is_freeze_clock:true}) [2024-09-26 16:00:35.623826] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337635623816, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623837] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c42c40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3730526293242960203, dag_cnt=28, dag_type_cnts=27) [2024-09-26 16:00:35.623839] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c42c40) [2024-09-26 16:00:35.623841] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337635623841, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405519, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101065}, checkpoint_unit_ptr:0x7f92a1a7db40}, is_freeze_clock:true}) [2024-09-26 16:00:35.623844] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337635623834, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623852] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c43170, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=4663792628280492528, dag_cnt=29, dag_type_cnts=28) [2024-09-26 16:00:35.623854] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c43170) [2024-09-26 16:00:35.623855] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337635623855, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405536, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:333}, checkpoint_unit_ptr:0x7f92a3bc4850}, is_freeze_clock:true}) [2024-09-26 16:00:35.623859] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337635623850, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623870] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c436a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8399557287176188414, dag_cnt=30, dag_type_cnts=29) [2024-09-26 16:00:35.623872] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c436a0) [2024-09-26 16:00:35.623874] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337635623874, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405552, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101066}, checkpoint_unit_ptr:0x7f92a3bc5020}, is_freeze_clock:true}) [2024-09-26 16:00:35.623877] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337635623865, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623887] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c43bd0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8068938909778324349, dag_cnt=31, dag_type_cnts=30) [2024-09-26 16:00:35.623888] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264c43bd0) [2024-09-26 16:00:35.623899] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, create_flush_dag_time:1727337635623899, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405569, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:334}, checkpoint_unit_ptr:0x7f92a3bc57f0}, is_freeze_clock:true}) [2024-09-26 16:00:35.623903] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337635623883, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623916] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c44100, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7665372330263152188, dag_cnt=32, dag_type_cnts=31) [2024-09-26 16:00:35.623918] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c44100) [2024-09-26 16:00:35.623920] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, create_flush_dag_time:1727337635623919, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405586, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:451}, checkpoint_unit_ptr:0x7f92a3bc4080}, is_freeze_clock:true}) [2024-09-26 16:00:35.623923] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337635623912, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623931] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c44630, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6759548225246904317, dag_cnt=33, dag_type_cnts=32) [2024-09-26 16:00:35.623935] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c44630) [2024-09-26 16:00:35.623937] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, create_flush_dag_time:1727337635623937, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405602, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:378}, checkpoint_unit_ptr:0x7f92a1a7f2b0}, is_freeze_clock:true}) [2024-09-26 16:00:35.623939] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337635623929, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.623981] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c44b60, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8944614441376604790, dag_cnt=34, dag_type_cnts=33) [2024-09-26 16:00:35.624006] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=25] success to create and add dag(ret=0, dag=0x7f9264c44b60) [2024-09-26 16:00:35.624030] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, create_flush_dag_time:1727337635624029, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209456003, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:121}, checkpoint_unit_ptr:0x7f92a3bc5fc0}, is_freeze_clock:true}) [2024-09-26 16:00:35.624034] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337635623946, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624047] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264c45090, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=507863347806783454, dag_cnt=35, dag_type_cnts=34) [2024-09-26 16:00:35.624049] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c45090) [2024-09-26 16:00:35.624051] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, create_flush_dag_time:1727337635624051, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405633, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:215}, checkpoint_unit_ptr:0x7f92a3bc86d0}, is_freeze_clock:true}) [2024-09-26 16:00:35.624054] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337635624043, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624065] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c455c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1604282355669434429, dag_cnt=36, dag_type_cnts=35) [2024-09-26 16:00:35.624067] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c455c0) [2024-09-26 16:00:35.624069] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, create_flush_dag_time:1727337635624069, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405649, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:456}, checkpoint_unit_ptr:0x7f92a3bc6790}, is_freeze_clock:true}) [2024-09-26 16:00:35.624072] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337635624061, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624083] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c45af0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3553649416163308074, dag_cnt=37, dag_type_cnts=36) [2024-09-26 16:00:35.624086] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c45af0) [2024-09-26 16:00:35.624088] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337635624087, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405666, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:154}, checkpoint_unit_ptr:0x7f92a3bc6f60}, is_freeze_clock:true}) [2024-09-26 16:00:35.624091] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337635624078, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624145] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c46020, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8097406237500329300, dag_cnt=38, dag_type_cnts=37) [2024-09-26 16:00:35.624150] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c46020) [2024-09-26 16:00:35.624152] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337635624152, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405682, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101015}, checkpoint_unit_ptr:0x7f92a3bc7730}, is_freeze_clock:true}) [2024-09-26 16:00:35.624155] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337635624097, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624202] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=17] add dag success(dag=0x7f9264c46550, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3976350754940852513, dag_cnt=39, dag_type_cnts=38) [2024-09-26 16:00:35.624207] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] success to create and add dag(ret=0, dag=0x7f9264c46550) [2024-09-26 16:00:35.624209] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337635624209, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405699, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101016}, checkpoint_unit_ptr:0x7f92a3bc7f00}, is_freeze_clock:true}) [2024-09-26 16:00:35.624212] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337635624172, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624224] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c46a80, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8720304653345806796, dag_cnt=40, dag_type_cnts=39) [2024-09-26 16:00:35.624227] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c46a80) [2024-09-26 16:00:35.624229] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, create_flush_dag_time:1727337635624229, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405716, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101063}, checkpoint_unit_ptr:0x7f92a3bca610}, is_freeze_clock:true}) [2024-09-26 16:00:35.624232] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337635624219, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624264] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=27] add dag success(dag=0x7f9264c46fb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1830562711513853292, dag_cnt=41, dag_type_cnts=40) [2024-09-26 16:00:35.624266] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c46fb0) [2024-09-26 16:00:35.624268] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, create_flush_dag_time:1727337635624268, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405733, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:452}, checkpoint_unit_ptr:0x7f92a3bc9e40}, is_freeze_clock:true}) [2024-09-26 16:00:35.624271] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337635624259, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624303] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=28] add dag success(dag=0x7f9264c474e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1536604874905918360, dag_cnt=42, dag_type_cnts=41) [2024-09-26 16:00:35.624305] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c474e0) [2024-09-26 16:00:35.624307] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337635624306, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405750, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:328}, checkpoint_unit_ptr:0x7f92a3bc8ea0}, is_freeze_clock:true}) [2024-09-26 16:00:35.624339] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=31] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337635624299, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624416] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=18] add dag success(dag=0x7f9264c47a10, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6933628521298738662, dag_cnt=43, dag_type_cnts=42) [2024-09-26 16:00:35.624443] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=26] success to create and add dag(ret=0, dag=0x7f9264c47a10) [2024-09-26 16:00:35.624446] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337635624446, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405767, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:330}, checkpoint_unit_ptr:0x7f92a3bc9670}, is_freeze_clock:true}) [2024-09-26 16:00:35.624449] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337635624358, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624498] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=20] add dag success(dag=0x7f9264c47f40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1056199536334423564, dag_cnt=44, dag_type_cnts=43) [2024-09-26 16:00:35.624501] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c47f40) [2024-09-26 16:00:35.624503] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337635624502, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405784, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:329}, checkpoint_unit_ptr:0x7f92a3bcb5b0}, is_freeze_clock:true}) [2024-09-26 16:00:35.624506] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337635624469, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624540] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c48470, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7405463607918801432, dag_cnt=45, dag_type_cnts=44) [2024-09-26 16:00:35.624545] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] success to create and add dag(ret=0, dag=0x7f9264c48470) [2024-09-26 16:00:35.624548] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, create_flush_dag_time:1727337635624548, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405801, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:332}, checkpoint_unit_ptr:0x7f92a3bcade0}, is_freeze_clock:true}) [2024-09-26 16:00:35.624551] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337635624513, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624607] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c489a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6699723950958743903, dag_cnt=46, dag_type_cnts=45) [2024-09-26 16:00:35.624613] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c489a0) [2024-09-26 16:00:35.624616] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, create_flush_dag_time:1727337635624615, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405818, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101067}, checkpoint_unit_ptr:0x7f92a3bcbd80}, is_freeze_clock:true}) [2024-09-26 16:00:35.624619] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337635624558, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.624625] INFO [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:655) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] traversal_flush successfully(ls_->get_ls_id()={id:1}, flush_tasks=[{table:0x7f92a1a70850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a70080, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71020, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a717f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71fc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72790, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72f60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73730, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73f00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a746d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75670, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a74ea0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76610, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75e40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a775b0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76de0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a77d80, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7a490, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a79cc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a794f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78550, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78d20, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7ac60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7cba0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7b430, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7bc00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7c3d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7d370, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7e310, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7eae0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7db40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a3bc4850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}]) [2024-09-26 16:00:35.624713] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=85] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-09-26 16:00:35.630736] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:35.630760] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=18] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.632503] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:35.632511] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.633511] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.633526] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=14] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337635623058, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:35.633533] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.633662] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=15] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:35.633670] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=7] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.633680] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847223}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:35.633757] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=39] succeed to build merge ctx(tablet_id={id:1152921504606847223}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847223}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635623166, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=10.51ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.634063] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=28] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:35.634786] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.634801] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=7] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=7999, total_rows=55993) [2024-09-26 16:00:35.634802] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=15] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337635622724, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:35.634810] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.634899] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:35.634918] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=17] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.634925] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847222}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:35.634947] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:1152921504606847222}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847222}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635623163, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=11.76ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.635185] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=23] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:35.635229] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61A-0-0, dag_ret:0, dag_status:2, start_time:1727337635623143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a43470, output_block_cnt_arr:0x7f9263a43478, concurrent_cnt:1, estimate_row_cnt:332829, estimate_occupy_size:150994944, latest_update_ts:1727337635623166, estimated_finish_time:1727337652635300}) [2024-09-26 16:00:35.635243] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f9264c3b8b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61A-0-0, dag_ret:0, dag_status:2, start_time:1727337635623143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.635249] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=6] task finish process(ret=0, start_time=1727337635623163, end_time=1727337635635249, runtime=12086, *this={this:0x7f9264c3b8b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61A-0-0, dag_ret:0, dag_status:2, start_time:1727337635623143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.635299] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=6] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.635310] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=10] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337635622676, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=17, max_schema_version_on_memtable=1725120002010696) [2024-09-26 16:00:35.635316] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=6] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=17, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.635326] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337635623297, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.635342] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752755184, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828752755184, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.635359] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.635363] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.635369] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:226}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752755184, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.635390] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:226}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752755184, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:226}, table_count:1, [{i:0, table_key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}, merge_scn:{val:1726241777443612000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635623609, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=11.76ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.635413] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1887, total_rows=92463) [2024-09-26 16:00:35.635416] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.635417] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61B-0-0, dag_ret:0, dag_status:2, start_time:1727337635623356, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f480, output_block_cnt_arr:0x7f9264b1f488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:56623104, latest_update_ts:1727337635623609, estimated_finish_time:1727337652635444}) [2024-09-26 16:00:35.635425] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3bde0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61B-0-0, dag_ret:0, dag_status:2, start_time:1727337635623356, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:35.635422] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200489}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:19, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.635431] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=5] task finish process(ret=0, start_time=1727337635623606, end_time=1727337635635431, runtime=11825, *this={this:0x7f9264c3bde0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61B-0-0, dag_ret:0, dag_status:2, start_time:1727337635623356, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:35.635441] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:200489}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200489}, table_count:2, [{i:0, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635622762, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=12.66ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.635737] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1, storage_schema:{this:0x7f92a23a72b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.635750] INFO [STORAGE] prepare_tx_data_list (ob_tx_data_memtable.cpp:266) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=11] construct list more than once. skip this time(this=0x7f92a2294080) [2024-09-26 16:00:35.635763] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=43] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C618-0-0, dag_ret:0, dag_status:2, start_time:1727337635622731, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1460, output_block_cnt_arr:0x7f92728f1468, concurrent_cnt:1, estimate_row_cnt:254187, estimate_occupy_size:138412032, latest_update_ts:1727337635622762, estimated_finish_time:1727337652635829}) [2024-09-26 16:00:35.635773] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=10] succeed to init merge ctx(task={this:0x7f9264c3ae50, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C618-0-0, dag_ret:0, dag_status:2, start_time:1727337635622731, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.635779] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=6] task finish process(ret=0, start_time=1727337635622759, end_time=1727337635635779, runtime=13020, *this={this:0x7f9264c3ae50, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C618-0-0, dag_ret:0, dag_status:2, start_time:1727337635622731, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.635926] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=4] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=8366, total_rows=58562) [2024-09-26 16:00:35.636365] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C619-0-0, dag_ret:0, dag_status:2, start_time:1727337635623132, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad470, output_block_cnt_arr:0x7f92684ad478, concurrent_cnt:1, estimate_row_cnt:329211, estimate_occupy_size:150994944, latest_update_ts:1727337635623163, estimated_finish_time:1727337652636437}) [2024-09-26 16:00:35.636378] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f9264c3b380, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C619-0-0, dag_ret:0, dag_status:2, start_time:1727337635623132, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.636385] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=6] task finish process(ret=0, start_time=1727337635623160, end_time=1727337635636384, runtime=13224, *this={this:0x7f9264c3b380, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C619-0-0, dag_ret:0, dag_status:2, start_time:1727337635623132, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.636448] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=2] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.636457] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=5] schedule one task(task=0x7f9264c49260, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.636461] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=2] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.636464] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=1] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.636556] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:4, allocator:0x7f926367a7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:35.636555] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=226, range=0x7f92ab056200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fa060, count:5, allocator:0x7f92636327d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:35.636577] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=18] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:4, allocator:0x7f926367b558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3e350, local_datums:0x7f9263a3e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:35.636822] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:35.636885] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.636889] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.636896] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:35.636899] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:35.636901] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9264d62140, hex: 3230323430393131303133333438, cstr: 20240911013348},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9264d6214e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d620e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3080, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7221, data_checksum:4284205003, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265892030, buf_size:7221, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1004563202, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:00:35.636916] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=14][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:35.636917] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:35.636920] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:35.636921] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:35.636923] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f926367a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926367a118, limit_param:null, stmt_allocator:0x7f926367a7d8, allocator:0x7f926367a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:00:35.636984] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=61][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:35.636986] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f926367a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926367a118, limit_param:null, stmt_allocator:0x7f926367a7d8, allocator:0x7f926367a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3a350, local_datums:0x7f9263a3a350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:00:35.637017] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=31][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:35.637020] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=2] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.637032] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=8] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337635622724, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.637042] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:35.637054] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:35.637057] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:35.637059] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337635636499, end_time=1727337635637059, runtime=560, *this={this:0x7f9264c49260, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C619-0-0, dag_ret:0, dag_status:2, start_time:1727337635623132, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.637065] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c49260, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C619-0-0, dag_ret:0, dag_status:2, start_time:1727337635623132, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:35.637074] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=13942, dag_cnt=45, dag_cnts_[dag.get_type()]=44, &dag=0x7f9264c3afc0, dag={ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C619-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635623132, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.637084] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C619-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337635623111, task_id:YB42C0A8002D-0006230056D8C619-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.637114] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337635637111, task_id:YB42C0A8002D-0006230056D8C61C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.637117] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=3] schedule one task(task=0x7f9264c3c310, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.640769] INFO [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:329) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=2] pre-process commit versions row(get_ls_id()={id:1}, do_recycle_=true, current_time=1727337635640768, prev_recycle_time=0) [2024-09-26 16:00:35.640851] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.640858] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.640861] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.640867] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.640953] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:3, allocator:0x7f92a339e7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3a320, local_datums:0x7f9263a3a320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:35.640971] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=15] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:3, allocator:0x7f92a339f558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:35.641155] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=7][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:35.641208] ERROR issue_dba_error (ob_log.cpp:1875) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.641211] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.641219] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:35.641221] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:35.641223] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62070, hex: 9F094D0300000000, int: 55380383},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d620a8, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d620e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3044, column_count:19, rowkey_column_count:3, has_column_checksum:0, row_count:83, row_store_type:0, opt:5, var_column_count:0, row_offset:16109, original_length:16381, max_merged_trans_version:1726163995623997000, data_length:16381, data_zlength:7759, data_checksum:3416849541, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9264dce030, buf_size:7759, data_size:16381, row_count:83, column_count:19, max_merged_trans_version:1726163995623997000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3537734793, row_count_delta:83, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16381}) [2024-09-26 16:00:35.641236] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=12][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:35.641238] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:35.641240] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:35.641241] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:35.641243] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f92a339ee98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a339ee98, limit_param:null, stmt_allocator:0x7f92a339f558, allocator:0x7f92a339f558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320133, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337635622676, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}) [2024-09-26 16:00:35.641310] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=67][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:35.641311] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f92a339ee98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a339ee98, limit_param:null, stmt_allocator:0x7f92a339f558, allocator:0x7f92a339f558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320133, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337635622676, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}]) [2024-09-26 16:00:35.641348] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=37][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:35.641351] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=2] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.641363] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=8] partition merge iter row count(i=1, row_count=83, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, table={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337635622676, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.641372] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:35.641382] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:35.641385] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:35.641387] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337635640905, end_time=1727337635641387, runtime=482, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C618-0-0, dag_ret:0, dag_status:2, start_time:1727337635622731, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.641394] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C618-0-0, dag_ret:0, dag_status:2, start_time:1727337635622731, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:35.641401] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=18670, dag_cnt=44, dag_cnts_[dag.get_type()]=43, &dag=0x7f9264c3aa90, dag={ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C618-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635622731, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.641411] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C618-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337635622720, task_id:YB42C0A8002D-0006230056D8C618-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.641508] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665b0060, count:4, allocator:0x7f9275d787d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665b0350, local_datums:0x7f92665b0350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:35.641523] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=12] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684e8060, count:4, allocator:0x7f9275d79558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684e8350, local_datums:0x7f92684e8350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:35.641769] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=6][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:35.641811] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.641813] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.641821] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:35.641823] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:35.641825] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9268432140, hex: 3230323430393131303133333131, cstr: 20240911013311},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f926843214e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684320e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9268432118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-15038, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7227, data_checksum:1452123993, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f926846c030, buf_size:7227, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1818264584, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:00:35.641836] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:35.641838] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:35.641839] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:35.641841] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:35.641843] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665b0350, local_datums:0x7f92665b0350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b03d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b0408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665b0350, local_datums:0x7f92665b0350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b03d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b0408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9275d78118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9275d78118, limit_param:null, stmt_allocator:0x7f9275d787d8, allocator:0x7f9275d787d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665b0350, local_datums:0x7f92665b0350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b03d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b0408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:00:35.641890] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=47][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:35.641891] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665b0350, local_datums:0x7f92665b0350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b03d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b0408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9275d78118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9275d78118, limit_param:null, stmt_allocator:0x7f9275d787d8, allocator:0x7f9275d787d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665b0350, local_datums:0x7f92665b0350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b03d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b0408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:00:35.641921] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=30][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:35.641923] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=1] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.641932] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=7] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337635623058, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.641941] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:35.641951] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:35.641954] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:35.641955] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337635641446, end_time=1727337635641955, runtime=509, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61A-0-0, dag_ret:0, dag_status:2, start_time:1727337635623143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.641962] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61A-0-0, dag_ret:0, dag_status:2, start_time:1727337635623143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:35.641969] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=18826, dag_cnt=43, dag_cnts_[dag.get_type()]=42, &dag=0x7f9264c3b4f0, dag={ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635623143, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.641977] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61A-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635623139, task_id:YB42C0A8002D-0006230056D8C61A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.642032] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726158220122116000) [2024-09-26 16:00:35.642048] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.642121] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337635642119, task_id:YB42C0A8002D-0006230056D8C61D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.642126] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=4] schedule one task(task=0x7f9264c3c840, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.642134] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337635642133, task_id:YB42C0A8002D-0006230056D8C61E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.642137] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=3] schedule one task(task=0x7f9264c3cd70, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.642158] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.642162] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.642164] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.642169] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.642173] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8621da0, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:480, tablet_addr:[816017-1849-0-2305843619101044736-1](offset=1994752,size=568,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:200004}, data_tablet_id:{id:200004}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1082501607, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1710828761605424, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1710828761605424}, table_store_addr:{ptr:{this:0x7f92b8622434, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844140939550720-0](offset=1974272,size=1054,type=2,seq=0), type:"MAJOR", tablet_id:{id:200004}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077597769620000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843906863834142-0](offset=1975326,size=836,type=2,seq=0), type:"MINOR", tablet_id:{id:200004}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178584585241000, upper_trans_version:1726178584585241000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337635623381, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697961984-1](offset=1982464,size=332,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834499404-0](offset=1982796,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843252955070940-0](offset=1982940,size=227,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828761613515000, v:0}, create_commit_version:1710828761613515000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a74ea0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:35.642215] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=42] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.642222] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.642988] INFO [STORAGE] get_ls_min_end_scn (ob_ls_tablet_service.cpp:5618) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=12] get ls min end scn finish(ls_checkpoint={val:1726163586617842000, v:0}) [2024-09-26 16:00:35.643005] WDIAG [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:761) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=10][errcode=-4023] failed to get snapshot version(ret=-4023, MTL_ID()=1) [2024-09-26 16:00:35.643010] INFO [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:773) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=4] get tx data recycle scn finish.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, recycle_scn={val:1726076389100953000, v:0}, min_end_scn_from_old_tablets={val:4611686018427387903, v:0}, min_end_scn_from_latest_tablets={val:1726076389100953000, v:0}) [2024-09-26 16:00:35.643042] WDIAG [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:336) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=31][errcode=-4023] get recycle ts failed.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:35.643048] WDIAG [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:236) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=3][errcode=-4023] process commit version row failed.(ret=-4023, ret="OB_EAGAIN", this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337635622515, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:35.643060] INFO [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:254) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=11] [TX DATA MERGE]pre process for merge done(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}, this=0x7f92a2294080, tg=time guard have no click events for optimization, fake_tx_data_guard={tx_data:{tx_id:{txid:9223372036854775807}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337635622515, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:35.643074] WDIAG [STORAGE.COMPACTION] pre_process_tx_data_table_merge_ (ob_tx_table_merge_task.cpp:117) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=13][errcode=-4023] do pre process for tx data table merge failed.(ret=-4023, ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, table={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337635622515, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:35.643089] WDIAG [STORAGE.COMPACTION] inner_init_ctx (ob_tx_table_merge_task.cpp:157) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=14][errcode=-4023] pre process tx data table for merge failed.(ret=-4023, ret="OB_EAGAIN", ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.643093] WDIAG [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1061) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=3][errcode=-4023] fail to inner init ctx(ret=-4023, tablet_id={id:49402}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, create_snapshot_version:0, is_full_merge:false, merge_level:1, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:4, range_array:[], concurrent_cnt:0, is_inited:false}, schema_ctx:{schema_version:1, storage_schema:{this:0x7f92a23a72b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:49402}, table_count:1, [{i:0, table_key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref:4}]}, schedule_major:false, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}, merge_scn:{val:4611686018427387903, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:NULL, compaction_filter:NULL, time_guard:total=20.11ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}) [2024-09-26 16:00:35.643117] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:975) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=24][errcode=-4023] failed to build merge ctx(ret=-4023, ctx->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.643120] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:994) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=3][errcode=-4023] sstable merge finish(ret=-4023, ctx=0x7f92a23a6060, task={this:0x7f9264c3a3f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C616-0-0, dag_ret:0, dag_status:2, start_time:1727337635622606, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:35.643127] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=7][errcode=-4023] failed to process task(ret=-4023) [2024-09-26 16:00:35.643130] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=2] task finish process(ret=-4023, start_time=1727337635622973, end_time=1727337635643129, runtime=20156, *this={this:0x7f9264c3a3f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C616-0-0, dag_ret:0, dag_status:2, start_time:1727337635622606, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:35.643136] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=5][errcode=-4023] failed to do work(ret=-4023, *task_={this:0x7f9264c3a3f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C616-0-0, dag_ret:0, dag_status:2, start_time:1727337635622606, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:35.643143] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=5] dag finished(dag_ret=-4023, runtime=20537, dag_cnt=42, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264c3a030, dag={ObIDag:{this:0x7f9264c3a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C616-0-0, dag_ret:-4023, dag_status:5, start_time:1727337635622606, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}) [2024-09-26 16:00:35.643152] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C616-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337635622560, task_id:YB42C0A8002D-0006230056D8C616-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.643187] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.643193] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.643195] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.643187] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635643184, task_id:YB42C0A8002D-0006230056D8C61F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.643200] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.643203] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=15] schedule one task(task=0x7f9264c3d2a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.643295] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=20][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:35.643356] ERROR issue_dba_error (ob_log.cpp:1875) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.643360] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.643367] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:35.643370] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:35.643372] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee0a8, hex: 0000000000000000, int: 0},idx=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92664ee0e0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee118, hex: 58A3C56A4A280BE8, int: -1726241732082425000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-19590, column_count:10, rowkey_column_count:5, has_column_checksum:0, row_count:224, row_store_type:0, opt:0, var_column_count:0, row_offset:15527, original_length:16363, max_merged_trans_version:1726241777443612000, data_length:16363, data_zlength:16363, data_checksum:2057046089, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6e04090, buf_size:16363, data_size:16363, row_count:224, column_count:10, max_merged_trans_version:1726241777443612000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1006468742, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16363}) [2024-09-26 16:00:35.643410] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=37][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:35.643412] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:35.643414] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:35.643415] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:35.643417] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:35.643419] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:35.643420] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fa3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fa3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f9263632118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263632118, limit_param:null, stmt_allocator:0x7f92636327d8, allocator:0x7f92636327d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1073741824, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fa3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337635623297, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}) [2024-09-26 16:00:35.643508] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=88][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:35.643511] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=3][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fa3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f9263632118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263632118, limit_param:null, stmt_allocator:0x7f92636327d8, allocator:0x7f92636327d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1073741824, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fa3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337635623297, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}]) [2024-09-26 16:00:35.643549] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=38][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:35.643551] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=1] partition merge iter row count(i=0, row_count=224, ghost_row_count=0, pkey={tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, table={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337635623297, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.643564] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:35.643585] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:35.643590] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:35.643594] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=3] task finish process(ret=-4184, start_time=1727337635636490, end_time=1727337635643594, runtime=7104, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61B-0-0, dag_ret:0, dag_status:2, start_time:1727337635623356, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:35.643601] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61B-0-0, dag_ret:0, dag_status:2, start_time:1727337635623356, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:35.643610] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=20254, dag_cnt=41, dag_cnts_[dag.get_type()]=41, &dag=0x7f9264c3ba20, dag={ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635623356, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}) [2024-09-26 16:00:35.643618] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635623343, task_id:YB42C0A8002D-0006230056D8C61B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.643656] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.643659] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.643662] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.643666] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.643670] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.643679] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241621541056000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.643889] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337635643885, task_id:YB42C0A8002D-0006230056D8C620-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.643896] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=6] schedule one task(task=0x7f9264c3d7d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.643921] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.643925] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.643927] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.643932] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.643936] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.643945] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.647793] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.647812] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=18] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337635622633, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=18, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:35.647821] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.648018] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:35.648024] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.648033] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200240}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.648056] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:200240}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200240}, table_count:2, [{i:0, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8637b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635623026, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=25.00ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.648279] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=24] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:35.648687] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C617-0-0, dag_ret:0, dag_status:2, start_time:1727337635622670, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505460, output_block_cnt_arr:0x7f9266505468, concurrent_cnt:1, estimate_row_cnt:85417, estimate_occupy_size:134217728, latest_update_ts:1727337635623026, estimated_finish_time:1727337652648750}) [2024-09-26 16:00:35.648699] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=11] succeed to init merge ctx(task={this:0x7f9264c3a920, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C617-0-0, dag_ret:0, dag_status:2, start_time:1727337635622670, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.648707] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=7] task finish process(ret=0, start_time=1727337635623024, end_time=1727337635648707, runtime=25683, *this={this:0x7f9264c3a920, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C617-0-0, dag_ret:0, dag_status:2, start_time:1727337635622670, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.648749] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=4] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.648873] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:3, allocator:0x7f92865507d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:35.648898] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=21] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c74060, count:3, allocator:0x7f9286551558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9262c74320, local_datums:0x7f9262c74320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:35.649146] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=24][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:35.649196] ERROR issue_dba_error (ob_log.cpp:1875) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.649199] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf2041c8 0xf217c49 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.649210] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=11][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:35.649213] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:35.649215] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92665b0108, hex: 313030303030333830, cstr: 100000380},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b00a8, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b00e0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-29413, column_count:20, rowkey_column_count:3, has_column_checksum:0, row_count:100, row_store_type:0, opt:0, var_column_count:0, row_offset:16082, original_length:16422, max_merged_trans_version:1726189672100745000, data_length:16454, data_zlength:5272, data_checksum:4171874375, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9268592030, buf_size:5272, data_size:16454, row_count:100, column_count:20, max_merged_trans_version:1726189672100745000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:371002229, row_count_delta:22, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16422}) [2024-09-26 16:00:35.649238] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=23][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:35.649241] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:35.649242] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:35.649244] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:35.649245] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1526) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, partition_fuser_->get_result_row()={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92684b2148, local_datums:0x7f92684b2148}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2190, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b21c8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b23c0, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b23f8, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, fuse_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b45d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9286550118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286550118, limit_param:null, stmt_allocator:0x7f92865507d8, allocator:0x7f92865507d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b45d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9262c74320, local_datums:0x7f9262c74320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c74368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c743a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c74598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c745d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9286550e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286550e98, limit_param:null, stmt_allocator:0x7f9286551558, allocator:0x7f9286551558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:7881706611452047458, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9262c74320, local_datums:0x7f9262c74320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c74368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c743a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c74598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c745d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337635622633, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:35.649371] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=126][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b45d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9286550118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286550118, limit_param:null, stmt_allocator:0x7f92865507d8, allocator:0x7f92865507d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b45d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9262c74320, local_datums:0x7f9262c74320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c74368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c743a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c74598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c745d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9286550e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286550e98, limit_param:null, stmt_allocator:0x7f9286551558, allocator:0x7f9286551558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:7881706611452047458, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9262c74320, local_datums:0x7f9262c74320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c74368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c743a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c74598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c745d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337635622633, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:35.649468] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=96][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:35.649471] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=2] partition merge iter row count(i=0, row_count=23, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.649483] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=8] partition merge iter row count(i=1, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337635622633, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.649492] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=6][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:35.649502] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:35.649506] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:35.649507] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337635648822, end_time=1727337635649507, runtime=685, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C617-0-0, dag_ret:0, dag_status:2, start_time:1727337635622670, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.649514] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C617-0-0, dag_ret:0, dag_status:2, start_time:1727337635622670, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:35.649522] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=26852, dag_cnt=40, dag_cnts_[dag.get_type()]=40, &dag=0x7f9264c3a560, dag={ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C617-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635622670, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.649522] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:35.649530] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C617-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337635622656, task_id:YB42C0A8002D-0006230056D8C617-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.649537] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=8] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.649563] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337635649560, task_id:YB42C0A8002D-0006230056D8C621-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.649567] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:35.649568] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=5] schedule one task(task=0x7f9264c3dd00, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.649586] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.649611] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.649614] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.649616] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.649620] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.649623] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.649628] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.650390] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337635623415, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.650406] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:0, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.650422] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.650426] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.650432] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:0, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.650432] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337635623381, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.650451] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=19] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828761605424, storage_schema:{this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828761605424, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.650467] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.650452] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:1}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:0, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1}, table_count:1, [{i:0, table_key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}, merge_scn:{val:1726241621541056000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635643650, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=6.78ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.650471] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.650477] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61F-0-0, dag_ret:0, dag_status:2, start_time:1727337635643203, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab480, output_block_cnt_arr:0x7f92684ab488, concurrent_cnt:1, estimate_row_cnt:3, estimate_occupy_size:6291456, latest_update_ts:1727337635643650, estimated_finish_time:1727337652650480}) [2024-09-26 16:00:35.650484] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3d2a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61F-0-0, dag_ret:0, dag_status:2, start_time:1727337635643203, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:35.650479] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200004}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1710828761605424, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.650490] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=5] task finish process(ret=0, start_time=1727337635643648, end_time=1727337635650490, runtime=6842, *this={this:0x7f9264c3d2a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61F-0-0, dag_ret:0, dag_status:2, start_time:1727337635643203, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:35.650515] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:200004}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828761605424, storage_schema:{this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200004}, table_count:1, [{i:0, table_key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8621da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635642151, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=8.32ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.650550] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=28] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61E-0-0, dag_ret:0, dag_status:2, start_time:1727337635642137, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92664ed460, output_block_cnt_arr:0x7f92664ed468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337635642151, estimated_finish_time:1727337652650553}) [2024-09-26 16:00:35.650557] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3cd70, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61E-0-0, dag_ret:0, dag_status:2, start_time:1727337635642137, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.650563] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=5] task finish process(ret=0, start_time=1727337635642149, end_time=1727337635650563, runtime=8414, *this={this:0x7f9264c3cd70, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61E-0-0, dag_ret:0, dag_status:2, start_time:1727337635642137, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.650583] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337635623437, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.650592] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753247280, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828753247280, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.650602] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.650605] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.650609] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:412}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753247280, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.650623] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:412}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753247280, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:412}, table_count:1, [{i:0, table_key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635643915, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=6.69ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.650640] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C620-0-0, dag_ret:0, dag_status:2, start_time:1727337635643896, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad470, output_block_cnt_arr:0x7f92684ad478, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337635643915, estimated_finish_time:1727337652650643}) [2024-09-26 16:00:35.650645] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c3d7d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C620-0-0, dag_ret:0, dag_status:2, start_time:1727337635643896, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.650649] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=4] task finish process(ret=0, start_time=1727337635643912, end_time=1727337635650649, runtime=6737, *this={this:0x7f9264c3d7d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C620-0-0, dag_ret:0, dag_status:2, start_time:1727337635643896, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.650690] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=4] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.650698] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=5] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.650701] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2] schedule one task(task=0x7f9262cae030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.650752] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=412, range=0x7f92a9cd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:7}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c74060, count:4, allocator:0x7f92bc9047d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9262c74350, local_datums:0x7f9262c74350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:35.650758] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=24] multi version scan iterator init succ(param.table_id_=200004, range=0x7f92aaad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:3, allocator:0x7f92bc9067d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:35.650780] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=16][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:35.650822] ERROR issue_dba_error (ob_log.cpp:1875) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.650825] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.650832] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:35.650834] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:35.650836] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665ae070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 23, flag: 0, null: 0, ptr: 0x7f92665ae140, hex: 6865617274626561745F736572766963655F65706F6368, cstr: heartbeat_service_epoch},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665ae0e0, hex: 8808AF9C9C5C0BE8, int: -1726184204453083000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665ae118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-7627, column_count:7, rowkey_column_count:4, has_column_checksum:0, row_count:1, row_store_type:0, opt:5, var_column_count:0, row_offset:136, original_length:80, max_merged_trans_version:1726184204453083000, data_length:80, data_zlength:80, data_checksum:1549639079, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f926f204090, buf_size:80, data_size:80, row_count:1, column_count:7, max_merged_trans_version:1726184204453083000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:37835160, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:80}) [2024-09-26 16:00:35.650847] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=11][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:35.650849] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:00:35.650851] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:00:35.650853] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:00:35.650854] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=1][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:00:35.650856] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337635623437, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.650870] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=12][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:35.650878] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:35.650882] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:35.650883] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337635650712, end_time=1727337635650883, runtime=171, *this={this:0x7f9262cae030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C620-0-0, dag_ret:0, dag_status:2, start_time:1727337635643896, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.650889] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262cae030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C620-0-0, dag_ret:0, dag_status:2, start_time:1727337635643896, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:35.650914] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=22] dag finished(dag_ret=-4184, runtime=7018, dag_cnt=39, dag_cnts_[dag.get_type()]=39, &dag=0x7f9264c3d410, dag={ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C620-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635643896, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.650922] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C620-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337635643885, task_id:YB42C0A8002D-0006230056D8C620-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.650928] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:35.650968] ERROR issue_dba_error (ob_log.cpp:1875) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.650971] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.650976] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=5][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:35.650978] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:35.650980] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e0a8, hex: C852A9F488570BE8, int: -1726189786434547000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e0e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19071, column_count:8, rowkey_column_count:3, has_column_checksum:0, row_count:218, row_store_type:0, opt:0, var_column_count:0, row_offset:15549, original_length:16361, max_merged_trans_version:1726190661653507000, data_length:16361, data_zlength:5996, data_checksum:2604580500, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9264d58030, buf_size:5996, data_size:16361, row_count:218, column_count:8, max_merged_trans_version:1726190661653507000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:83488362, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16361}) [2024-09-26 16:00:35.650990] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=10][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:35.650992] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:35.650993] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:35.650995] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:35.650996] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f92bc906118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92bc906118, limit_param:null, stmt_allocator:0x7f92bc9067d8, allocator:0x7f92bc9067d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337635623381, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:35.651047] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=51][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:35.651049] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f92bc906118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92bc906118, limit_param:null, stmt_allocator:0x7f92bc9067d8, allocator:0x7f92bc9067d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337635623381, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:35.651081] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=32][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:35.651084] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=2] partition merge iter row count(i=0, row_count=218, ghost_row_count=0, pkey={tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337635623381, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.651095] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:35.651103] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:35.651106] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:35.651108] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337635650713, end_time=1727337635651108, runtime=395, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61E-0-0, dag_ret:0, dag_status:2, start_time:1727337635642137, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.651114] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61E-0-0, dag_ret:0, dag_status:2, start_time:1727337635642137, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:35.651121] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=8984, dag_cnt=38, dag_cnts_[dag.get_type()]=38, &dag=0x7f9264c3c9b0, dag={ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635642137, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.651129] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61E-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635642133, task_id:YB42C0A8002D-0006230056D8C61E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.651184] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=1, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:8}, cols_desc:{ObIArray:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:5, allocator:0x7f92636627d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:35.651250] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635651246, task_id:YB42C0A8002D-0006230056D8C622-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.651256] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=5] schedule one task(task=0x7f9264c3e230, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.651262] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337635651261, task_id:YB42C0A8002D-0006230056D8C623-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.651264] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=2] schedule one task(task=0x7f9264c3e760, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.651287] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.651291] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.651293] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.651299] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.651302] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.651309] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241316899724000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.651328] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.651330] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.651332] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.651335] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.651337] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.651341] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241465393532000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.651555] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=11][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:35.651591] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.651593] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.651599] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:35.651601] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:35.651603] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 17, flag: 0, null: 0, ptr: 0x7f9263a3e178, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e0a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 15, flag: 0, null: 0, ptr: 0x7f9263a3e189, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e118, hex: 60D2640CCC290BE8, int: -1726240075808452000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-20032, column_count:8, rowkey_column_count:5, has_column_checksum:0, row_count:158, row_store_type:0, opt:0, var_column_count:0, row_offset:15849, original_length:16421, max_merged_trans_version:1726241621541056000, data_length:16421, data_zlength:16421, data_checksum:3599888316, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6804090, buf_size:16421, data_size:16421, row_count:158, column_count:8, max_merged_trans_version:1726241621541056000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1714289658, row_count_delta:2, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16421}) [2024-09-26 16:00:35.651613] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=10][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:35.651614] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=1][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:35.651616] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:35.651617] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:35.651618] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:35.651620] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:35.651621] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f9263662118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263662118, limit_param:null, stmt_allocator:0x7f92636627d8, allocator:0x7f92636627d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337635623415, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:35.651681] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=60][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:35.651683] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f9263662118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263662118, limit_param:null, stmt_allocator:0x7f92636627d8, allocator:0x7f92636627d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263a3a360, local_datums:0x7f9263a3a360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337635623415, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:35.651718] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=35][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:35.651720] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=1] partition merge iter row count(i=0, row_count=158, ghost_row_count=0, pkey={tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, table={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337635623415, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.651730] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:35.651737] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:35.651740] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:35.651741] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337635651154, end_time=1727337635651741, runtime=587, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61F-0-0, dag_ret:0, dag_status:2, start_time:1727337635643203, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:35.651747] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61F-0-0, dag_ret:0, dag_status:2, start_time:1727337635643203, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:35.651753] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=8550, dag_cnt=37, dag_cnts_[dag.get_type()]=37, &dag=0x7f9264c3cee0, dag={ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635643203, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}) [2024-09-26 16:00:35.651773] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61F-0-0] [lt=18] succeed to del sys task(removed_task={start_time:1727337635643184, task_id:YB42C0A8002D-0006230056D8C61F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.651801] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337635623344, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.651810] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=8] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:35.651813] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92728f1480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.651841] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635651838, task_id:YB42C0A8002D-0006230056D8C624-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.651848] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=6] schedule one task(task=0x7f9264c3ec90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.651879] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.651885] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.651887] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.651892] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.651895] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.651902] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.651937] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3223, total_rows=22561) [2024-09-26 16:00:35.651942] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.651952] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=6] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847409}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.651970] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:1152921504606847409}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92728f1480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847409}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8640e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635643180, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=8.77ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.652080] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61D-0-0, dag_ret:0, dag_status:2, start_time:1727337635642125, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f15d0, output_block_cnt_arr:0x7f92728f15d8, concurrent_cnt:1, estimate_row_cnt:32439, estimate_occupy_size:18874368, latest_update_ts:1727337635643180, estimated_finish_time:1727337652652088}) [2024-09-26 16:00:35.652087] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3c840, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61D-0-0, dag_ret:0, dag_status:2, start_time:1727337635642125, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.652092] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=4] task finish process(ret=0, start_time=1727337635643178, end_time=1727337635652092, runtime=8914, *this={this:0x7f9264c3c840, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61D-0-0, dag_ret:0, dag_status:2, start_time:1727337635642125, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.652138] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337635623320, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.652153] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=14] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:35.652156] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92665ff480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.652271] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3214, total_rows=22498) [2024-09-26 16:00:35.652275] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.652282] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92665ff480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847408}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.652301] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:1152921504606847408}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92665ff480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847408}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635640837, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=11.44ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.652411] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61C-0-0, dag_ret:0, dag_status:2, start_time:1727337635637117, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665ff5d0, output_block_cnt_arr:0x7f92665ff5d8, concurrent_cnt:1, estimate_row_cnt:35098, estimate_occupy_size:18874368, latest_update_ts:1727337635640837, estimated_finish_time:1727337652652419}) [2024-09-26 16:00:35.652419] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264c3c310, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61C-0-0, dag_ret:0, dag_status:2, start_time:1727337635637117, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.652425] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=5] task finish process(ret=0, start_time=1727337635640833, end_time=1727337635652425, runtime=11592, *this={this:0x7f9264c3c310, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61C-0-0, dag_ret:0, dag_status:2, start_time:1727337635637117, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.652443] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=3] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.652450] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=3] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.652575] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1152921504606847409, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:5, allocator:0x7f92c4a5e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:35.652745] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337635623478, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.652762] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753327496, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753327496, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.652774] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.652777] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.652782] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:444}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753327496, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.652800] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:444}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753327496, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:444}, table_count:1, [{i:0, table_key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8632d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635649605, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.17ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.652823] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C621-0-0, dag_ret:0, dag_status:2, start_time:1727337635649568, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505490, output_block_cnt_arr:0x7f9266505498, concurrent_cnt:1, estimate_row_cnt:4, estimate_occupy_size:6291456, latest_update_ts:1727337635649605, estimated_finish_time:1727337652652826}) [2024-09-26 16:00:35.652829] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c3dd00, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C621-0-0, dag_ret:0, dag_status:2, start_time:1727337635649568, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.652834] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=5] task finish process(ret=0, start_time=1727337635649604, end_time=1727337635652834, runtime=3230, *this={this:0x7f9264c3dd00, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C621-0-0, dag_ret:0, dag_status:2, start_time:1727337635649568, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.652892] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=2] schedule one task(task=0x7f9264c49260, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.652920] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=16][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:35.652961] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.652964] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.652970] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=5][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:35.652972] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:35.652974] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 7, flag: 0, null: 0, ptr: 0x7f9264d58178, hex: 31303033343330, cstr: 1003430},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9264d5817f, hex: 3230323430393133303435303537, cstr: 20240913045057},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d580e0, hex: 6E48500300000000, int: 55593070},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d58118, hex: 58B03BC5A1610BE8, int: -1726178684739801000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d58150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-26357, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:250, row_store_type:0, opt:5, var_column_count:0, row_offset:15394, original_length:16334, max_merged_trans_version:1726190643638412000, data_length:16334, data_zlength:5786, data_checksum:2846467531, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9264d5a030, buf_size:5786, data_size:16334, row_count:250, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:796410710, row_count_delta:250, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16334}) [2024-09-26 16:00:35.653027] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=53][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:35.653030] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=3][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:35.653031] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:35.653033] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:35.653034] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92c4a5e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c4a5e118, limit_param:null, stmt_allocator:0x7f92c4a5e7d8, allocator:0x7f92c4a5e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337635623344, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:35.653159] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=125][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:35.653256] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337635623523, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.653260] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=1152921504606847408, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:5, allocator:0x7f92864e07d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:35.653270] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.653284] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.653289] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.653162] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92c4a5e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c4a5e118, limit_param:null, stmt_allocator:0x7f92c4a5e7d8, allocator:0x7f92c4a5e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337635623344, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:35.653301] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=139][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:35.653295] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:331}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.653304] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=2] partition merge iter row count(i=0, row_count=250, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337635623344, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.653317] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:35.653326] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:35.653341] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=15][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:35.653343] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337635652539, end_time=1727337635653343, runtime=804, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61D-0-0, dag_ret:0, dag_status:2, start_time:1727337635642125, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.653313] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:331}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:331}, table_count:1, [{i:0, table_key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}, merge_scn:{val:1726241465393532000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8633ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635651325, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.97ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.653350] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61D-0-0, dag_ret:0, dag_status:2, start_time:1727337635642125, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:35.653358] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=11233, dag_cnt=36, dag_cnts_[dag.get_type()]=36, &dag=0x7f9264c3c480, dag={ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635642125, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.653361] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=39] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C623-0-0, dag_ret:0, dag_status:2, start_time:1727337635651264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad480, output_block_cnt_arr:0x7f92684ad488, concurrent_cnt:1, estimate_row_cnt:45, estimate_occupy_size:0, latest_update_ts:1727337635651325, estimated_finish_time:1727337652653361}) [2024-09-26 16:00:35.653366] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61D-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635642119, task_id:YB42C0A8002D-0006230056D8C61D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.653368] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3e760, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C623-0-0, dag_ret:0, dag_status:2, start_time:1727337635651264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:35.653384] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=15] task finish process(ret=0, start_time=1727337635651324, end_time=1727337635653384, runtime=2060, *this={this:0x7f9264c3e760, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C623-0-0, dag_ret:0, dag_status:2, start_time:1727337635651264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:35.653449] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337635623504, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.653459] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752918752, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828752918752, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.653470] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.653473] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.653465] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=444, range=0x7f92aab52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:6, allocator:0x7f92b32387d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264a62390, local_datums:0x7f9264a62390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:35.653477] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:309}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752918752, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.653496] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:309}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752918752, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:309}, table_count:1, [{i:0, table_key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}, merge_scn:{val:1726241316899724000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8631da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635651281, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.19ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.653516] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C622-0-0, dag_ret:0, dag_status:2, start_time:1727337635651255, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a7490, output_block_cnt_arr:0x7f92a23a7498, concurrent_cnt:1, estimate_row_cnt:6, estimate_occupy_size:6291456, latest_update_ts:1727337635651281, estimated_finish_time:1727337652653519}) [2024-09-26 16:00:35.653522] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c3e230, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C622-0-0, dag_ret:0, dag_status:2, start_time:1727337635651255, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:35.653527] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=5] task finish process(ret=0, start_time=1727337635651279, end_time=1727337635653527, runtime=2248, *this={this:0x7f9264c3e230, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C622-0-0, dag_ret:0, dag_status:2, start_time:1727337635651255, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:35.653542] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:35.653542] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=6] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.653549] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=4] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.653554] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635653552, task_id:YB42C0A8002D-0006230056D8C625-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.653572] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=3] schedule one task(task=0x7f9264c3f1c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.653602] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.653605] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.653612] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:35.653614] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:35.653616] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 12, flag: 0, null: 0, ptr: 0x7f92684de178, hex: 30323232362D32312D373825, cstr: 02226-21-78%},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f92684de184, hex: 3230323430393133303435343039, cstr: 20240913045409},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684de0e0, hex: CD7F500300000000, int: 55607245},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684de118, hex: E8EC05CCA0600BE8, int: -1726179788432479000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684de150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-11869, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:238, row_store_type:0, opt:5, var_column_count:0, row_offset:15429, original_length:16321, max_merged_trans_version:1726190643638412000, data_length:16321, data_zlength:5388, data_checksum:152456761, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92665a6030, buf_size:5388, data_size:16321, row_count:238, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2114797869, row_count_delta:238, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16321}) [2024-09-26 16:00:35.653630] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=14][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:35.653632] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:35.653634] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:35.653635] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:35.653653] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:35.653636] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92864e0118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92864e0118, limit_param:null, stmt_allocator:0x7f92864e07d8, allocator:0x7f92864e07d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337635623320, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:35.653687] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=51][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:35.653689] ERROR issue_dba_error (ob_log.cpp:1875) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.653691] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.653698] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:35.653700] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:35.653701] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d340a8, hex: 0300000000000000, int: 3},idx=2:{len: 12, flag: 0, null: 0, ptr: 0x7f9287d341b0, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34118, hex: 420B000000000000, int: 2882},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34150, hex: 288E681FE7280BE8, int: -1726241059036951000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d34188, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:24981, column_count:9, rowkey_column_count:6, has_column_checksum:0, row_count:207, row_store_type:0, opt:0, var_column_count:0, row_offset:15610, original_length:16378, max_merged_trans_version:1726241059036951000, data_length:16378, data_zlength:16378, data_checksum:1574137226, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6c04090, buf_size:16378, data_size:16378, row_count:207, column_count:9, max_merged_trans_version:1726241059036951000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:666643265, row_count_delta:3, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16378}) [2024-09-26 16:00:35.653712] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=11][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:35.653714] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:35.653716] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:35.653717] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:35.653688] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92864e0118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92864e0118, limit_param:null, stmt_allocator:0x7f92864e07d8, allocator:0x7f92864e07d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337635623320, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:35.653718] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:35.653720] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:35.653720] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=32][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:35.653722] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=2] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337635623320, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.653734] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=8][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.653743] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635653228, end_time=1727337635653743, runtime=515, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61C-0-0, dag_ret:0, dag_status:2, start_time:1727337635637117, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.653752] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=16635, dag_cnt=35, dag_cnts_[dag.get_type()]=35, &dag=0x7f9264c3bf50, dag={ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C61C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635637117, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.653760] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C61C-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635637111, task_id:YB42C0A8002D-0006230056D8C61C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.653721] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264a62390, local_datums:0x7f9264a62390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a624b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264a62390, local_datums:0x7f9264a62390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a624b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f92b3238118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b3238118, limit_param:null, stmt_allocator:0x7f92b32387d8, allocator:0x7f92b32387d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264a62390, local_datums:0x7f9264a62390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a624b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337635623478, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}) [2024-09-26 16:00:35.653773] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=52][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.653776] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=0] partition merge iter row count(i=0, row_count=207, ghost_row_count=0, pkey={tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337635623478, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.653794] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635653422, end_time=1727337635653794, runtime=372, *this={this:0x7f9264c49260, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C621-0-0, dag_ret:0, dag_status:2, start_time:1727337635649568, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.653803] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4235, dag_cnt=34, dag_cnts_[dag.get_type()]=34, &dag=0x7f9264c3d940, dag={ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C621-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635649568, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.653810] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C621-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635649560, task_id:YB42C0A8002D-0006230056D8C621-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.653817] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=309, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:11}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264b20060, count:7, allocator:0x7f92636627d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9264b203b0, local_datums:0x7f9264b203b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:35.653872] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.653876] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.653878] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.653884] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.653888] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b863cf80, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:522, tablet_addr:[816017-1849-0-2305843664198197248-2](offset=1990656,size=610,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:487}, data_tablet_id:{id:487}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1549633270, row_count:11693}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863d6c8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844647745695744-0](offset=1978368,size=1526,type=2,seq=0), type:"MAJOR", tablet_id:{id:487}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389060348000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843977730799094-0](offset=1979894,size=902,type=2,seq=0), type:"MINOR", tablet_id:{id:487}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178190844721000, upper_trans_version:1726178190844721000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337635623574, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd4850, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209506557}, this:0x7f92a3bd4850, timestamp:1727334209506557, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:66, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274231198751000, v:0}, rec_scn:{val:1726241813664082000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843367845449728-2](offset=1986560,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046758222-0](offset=1986894,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881333244-0](offset=1987068,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447853278-0](offset=1987294,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a79cc0, 0x7f92a3bd4850, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.653932] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=44] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.653939] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.654002] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=27][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.654036] ERROR issue_dba_error (ob_log.cpp:1875) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.654038] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.654048] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=0] partition merge iter row count(i=0, row_count=156, ghost_row_count=0, pkey={tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, table={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337635623504, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.654066] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635653784, end_time=1727337635654066, runtime=282, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C622-0-0, dag_ret:0, dag_status:2, start_time:1727337635651255, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:35.654074] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2819, dag_cnt=33, dag_cnts_[dag.get_type()]=33, &dag=0x7f9264c3de70, dag={ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C622-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635651255, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}) [2024-09-26 16:00:35.654081] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C622-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635651246, task_id:YB42C0A8002D-0006230056D8C622-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.654129] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=21] multi version scan iterator init succ(param.table_id_=331, range=0x7f92ab152200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287db8060, count:5, allocator:0x7f92b33d87d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9287db8360, local_datums:0x7f9287db8360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:35.654271] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:35.654288] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337635623542, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.654300] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753357664, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753357664, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.654323] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.654367] ERROR issue_dba_error (ob_log.cpp:1875) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.654371] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:35.654370] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.654376] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.654378] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=0] partition merge iter row count(i=0, row_count=196, ghost_row_count=0, pkey={tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, table={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337635623523, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.654382] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:458}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753357664, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.654407] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635654104, end_time=1727337635654407, runtime=303, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C623-0-0, dag_ret:0, dag_status:2, start_time:1727337635651264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:35.654417] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3152, dag_cnt=32, dag_cnts_[dag.get_type()]=32, &dag=0x7f9264c3e3a0, dag={ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C623-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635651264, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}) [2024-09-26 16:00:35.654423] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C623-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337635651261, task_id:YB42C0A8002D-0006230056D8C623-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.654414] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=25] succeed to build merge ctx(tablet_id={id:458}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753357664, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:458}, table_count:1, [{i:0, table_key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8627b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635651865, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.51ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.654452] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337635623574, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.654464] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753417144, storage_schema:{this:0x7f9265b05330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828753417144, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.654530] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=18] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:35.654543] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:35.654546] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.654552] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265b05330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:487}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753417144, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.654573] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:487}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753417144, storage_schema:{this:0x7f9265b05330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:487}, table_count:1, [{i:0, table_key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635653865, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=684us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.654696] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C624-0-0, dag_ret:0, dag_status:2, start_time:1727337635651847, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f4a0, output_block_cnt_arr:0x7f9264b1f4a8, concurrent_cnt:1, estimate_row_cnt:14598, estimate_occupy_size:4194304, latest_update_ts:1727337635651865, estimated_finish_time:1727337652654697}) [2024-09-26 16:00:35.654714] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=18] succeed to init merge ctx(task={this:0x7f9264c3ec90, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C624-0-0, dag_ret:0, dag_status:2, start_time:1727337635651847, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.654720] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=5] task finish process(ret=0, start_time=1727337635651862, end_time=1727337635654720, runtime=2858, *this={this:0x7f9264c3ec90, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C624-0-0, dag_ret:0, dag_status:2, start_time:1727337635651847, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.654732] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=18] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:35.654742] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=3] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.654759] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] succeed to add sys task(task={start_time:1727337635654757, task_id:YB42C0A8002D-0006230056D8C626-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.654763] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=3] schedule one task(task=0x7f9264c3f6f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.654768] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635654767, task_id:YB42C0A8002D-0006230056D8C627-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.654771] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=3] schedule one task(task=0x7f9264c3fc20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.654776] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635654775, task_id:YB42C0A8002D-0006230056D8C628-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.654778] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=2] schedule one task(task=0x7f9264c40150, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.654782] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635654781, task_id:YB42C0A8002D-0006230056D8C629-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.654784] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=1] schedule one task(task=0x7f9264c40680, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.654811] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.654815] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.654817] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.654822] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.654825] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.654832] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.654917] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C625-0-0, dag_ret:0, dag_status:2, start_time:1727337635653572, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265b054a0, output_block_cnt_arr:0x7f9265b054a8, concurrent_cnt:1, estimate_row_cnt:2199, estimate_occupy_size:0, latest_update_ts:1727337635653865, estimated_finish_time:1727337652654916}) [2024-09-26 16:00:35.654930] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=12] succeed to init merge ctx(task={this:0x7f9264c3f1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C625-0-0, dag_ret:0, dag_status:2, start_time:1727337635653572, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.654936] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=6] task finish process(ret=0, start_time=1727337635653863, end_time=1727337635654936, runtime=1073, *this={this:0x7f9264c3f1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C625-0-0, dag_ret:0, dag_status:2, start_time:1727337635653572, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.654986] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.654991] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.654993] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.654997] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.655001] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.655007] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.655096] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=458, range=0x7f92ab0d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:8, allocator:0x7f92bc9047d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9263a3a3e0, local_datums:0x7f9263a3a3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:35.655320] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=31][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.655344] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337635623637, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.655355] ERROR issue_dba_error (ob_log.cpp:1875) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.655358] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.655357] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753441792, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828753441792, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.655368] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=0] partition merge iter row count(i=0, row_count=247, ghost_row_count=0, pkey={tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337635623542, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.655389] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635655057, end_time=1727337635655389, runtime=332, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C624-0-0, dag_ret:0, dag_status:2, start_time:1727337635651847, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.655398] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3551, dag_cnt=31, dag_cnts_[dag.get_type()]=31, &dag=0x7f9264c3e8d0, dag={ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C624-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635651847, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.655405] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C624-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635651838, task_id:YB42C0A8002D-0006230056D8C624-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.655413] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1295, total_rows=1295) [2024-09-26 16:00:35.655416] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.655438] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.655442] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.655444] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.655448] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.655422] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:501}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753441792, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.655465] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=36] succeed to build merge ctx(tablet_id={id:501}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753441792, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:501}, table_count:1, [{i:0, table_key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635654799, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=620us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.655452] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861cf80, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:528, tablet_addr:[816017-1849-0-2305843670640488448-2](offset=1830912,size=616,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:455}, data_tablet_id:{id:455}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4242650745, row_count:15909}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861d6ec, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305845137371811840-0](offset=1822720,size=1982,type=2,seq=0), type:"MAJOR", tablet_id:{id:455}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389074974000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305845058988660670-0](offset=1824702,size=1909,type=2,seq=0), type:"MINOR", tablet_id:{id:455}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178192070487000, upper_trans_version:1726178192070487000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337635623598, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcec60, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209461810}, this:0x7f92a3bcec60, timestamp:1727334209461810, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:55, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274239678928000, v:0}, rec_scn:{val:1726241818022255001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771548160-2](offset=1826816,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843202489049421-0](offset=1827149,size=180,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881173505-0](offset=1827329,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447693539-0](offset=1827555,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a794f0, 0x7f92a3bcec60, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.655514] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.655521] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.655544] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.655546] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.655548] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.655551] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.655553] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.655550] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=31] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C628-0-0, dag_ret:0, dag_status:2, start_time:1727337635654778, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a43490, output_block_cnt_arr:0x7f9263a43498, concurrent_cnt:1, estimate_row_cnt:1295, estimate_occupy_size:0, latest_update_ts:1727337635654799, estimated_finish_time:1727337652655550}) [2024-09-26 16:00:35.655557] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.655557] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c40150, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C628-0-0, dag_ret:0, dag_status:2, start_time:1727337635654778, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.655562] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=5] task finish process(ret=0, start_time=1727337635654797, end_time=1727337635655562, runtime=765, *this={this:0x7f9264c40150, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C628-0-0, dag_ret:0, dag_status:2, start_time:1727337635654778, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.655597] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=1] schedule one task(task=0x7f9264ae8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.655612] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=4] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.655620] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337635655618, task_id:YB42C0A8002D-0006230056D8C62A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.655595] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337635623658, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.655631] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=3] schedule one task(task=0x7f9264c40bb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.655628] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=32] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753360568, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753360568, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.655648] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=16] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.655651] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.655657] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:472}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753360568, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.655693] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=501, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a36060, count:6, allocator:0x7f92c183a7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263a36390, local_datums:0x7f9263a36390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:35.655689] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=27] succeed to build merge ctx(tablet_id={id:472}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753360568, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:472}, table_count:1, [{i:0, table_key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635654962, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=692us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.655720] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=25] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C629-0-0, dag_ret:0, dag_status:2, start_time:1727337635654783, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505460, output_block_cnt_arr:0x7f9266505468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337635654962, estimated_finish_time:1727337652655720}) [2024-09-26 16:00:35.655727] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c40680, type:15, status:2, dag:{ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C629-0-0, dag_ret:0, dag_status:2, start_time:1727337635654783, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.655732] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=4] task finish process(ret=0, start_time=1727337635654959, end_time=1727337635655731, runtime=772, *this={this:0x7f9264c40680, type:15, status:2, dag:{ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C629-0-0, dag_ret:0, dag_status:2, start_time:1727337635654783, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.655767] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.655809] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.655812] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.655823] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=0] partition merge iter row count(i=0, row_count=56, ghost_row_count=0, pkey={tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337635623637, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.655822] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=487, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:11}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264dce060, count:8, allocator:0x7f92636827d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9264dce3e0, local_datums:0x7f9264dce3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:35.655841] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635655654, end_time=1727337635655841, runtime=187, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C628-0-0, dag_ret:0, dag_status:2, start_time:1727337635654778, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.655852] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=1] dag finished(dag_ret=-4184, runtime=1074, dag_cnt=30, dag_cnts_[dag.get_type()]=30, &dag=0x7f9264c3fd90, dag={ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C628-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635654778, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.655859] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C628-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635654775, task_id:YB42C0A8002D-0006230056D8C628-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.655888] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337635623598, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.655897] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753332736, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=36, max_schema_version_in_memtable=1710828753332736, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.656034] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:35.656037] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.656053] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.656043] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:455}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:10, rowkey_column_count:9, schema_rowkey_col_cnt:7, full_stored_col_cnt:38, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753332736, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=26 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.656065] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:455}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753332736, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:455}, table_count:1, [{i:0, table_key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635655433, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=608us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.656092] ERROR issue_dba_error (ob_log.cpp:1875) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.656095] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.656105] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=1] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337635623574, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.656127] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635655753, end_time=1727337635656127, runtime=374, *this={this:0x7f9264ae8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C625-0-0, dag_ret:0, dag_status:2, start_time:1727337635653572, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.656136] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2564, dag_cnt=29, dag_cnts_[dag.get_type()]=29, &dag=0x7f9264c3ee00, dag={ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C625-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635653572, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.656143] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C625-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635653552, task_id:YB42C0A8002D-0006230056D8C625-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.656170] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=2] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.656190] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=15] succeed to add sys task(task={start_time:1727337635656188, task_id:YB42C0A8002D-0006230056D8C62B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.656194] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=4] schedule one task(task=0x7f9264c410e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.656199] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635656198, task_id:YB42C0A8002D-0006230056D8C62C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.656199] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.656202] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=3] schedule one task(task=0x7f9264c41610, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.656203] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.656205] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726077706339704027) [2024-09-26 16:00:35.656209] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.656212] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.656218] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.656227] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=19] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:35.656266] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=472, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:3, allocator:0x7f92636327d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:35.656302] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.656333] ERROR issue_dba_error (ob_log.cpp:1875) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.656336] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.656344] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=0] partition merge iter row count(i=0, row_count=17, ghost_row_count=0, pkey={tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337635623658, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.656362] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635656236, end_time=1727337635656361, runtime=125, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C629-0-0, dag_ret:0, dag_status:2, start_time:1727337635654783, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.656370] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1586, dag_cnt=28, dag_cnts_[dag.get_type()]=28, &dag=0x7f9264c402c0, dag={ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C629-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635654783, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.656376] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C629-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337635654781, task_id:YB42C0A8002D-0006230056D8C629-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.656387] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C626-0-0, dag_ret:0, dag_status:2, start_time:1727337635654763, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a454b0, output_block_cnt_arr:0x7f9263a454b8, concurrent_cnt:1, estimate_row_cnt:3824, estimate_occupy_size:4194304, latest_update_ts:1727337635655433, estimated_finish_time:1727337652656388}) [2024-09-26 16:00:35.656396] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264c3f6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C626-0-0, dag_ret:0, dag_status:2, start_time:1727337635654763, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.656401] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=5] task finish process(ret=0, start_time=1727337635655431, end_time=1727337635656401, runtime=970, *this={this:0x7f9264c3f6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C626-0-0, dag_ret:0, dag_status:2, start_time:1727337635654763, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.656433] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.656436] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.656438] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.656442] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.656445] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8622d40, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:492, tablet_addr:[816017-1849-0-2305843631985766400-2](offset=1814528,size=580,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:140}, data_tablet_id:{id:140}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726073917016589000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2305242685, row_count:150}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86233d4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844315959259136-0](offset=1765376,size=1217,type=2,seq=0), type:"MAJOR", tablet_id:{id:140}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726070403667750001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843943370847425-0](offset=1766593,size=870,type=2,seq=0), type:"MINOR", tablet_id:{id:140}, scn_range:{start_scn:{val:1726073917016589000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726167395874925000, upper_trans_version:1726167395874925000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337635623714, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcfc00, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495712}, this:0x7f92a3bcfc00, timestamp:1727334209495712, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:57, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726256203821911000, v:0}, rec_scn:{val:1726241803398655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697781760-2](offset=1802240,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834319180-0](offset=1802572,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881148892-0](offset=1802716,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447668926-0](offset=1802942,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7b430, 0x7f92a3bcfc00, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.656509] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.656515] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.656537] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.656540] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.656541] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.656547] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.656549] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.656553] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.656572] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337635623618, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.656572] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337635623676, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.656582] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753001832, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=21, max_schema_version_in_memtable=1710828753001832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.656580] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753437040, storage_schema:{this:0x7f9265b01330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=57, max_schema_version_in_memtable=1710828753437040, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.656594] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.656597] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.656602] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:342}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:23, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753001832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.656619] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:342}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753001832, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:342}, table_count:1, [{i:0, table_key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635656186, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=413us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.656638] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1362, total_rows=1362) [2024-09-26 16:00:35.656641] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.656639] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62A-0-0, dag_ret:0, dag_status:2, start_time:1727337635655630, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a7490, output_block_cnt_arr:0x7f92a23a7498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337635656186, estimated_finish_time:1727337652656639}) [2024-09-26 16:00:35.656654] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=14] succeed to init merge ctx(task={this:0x7f9264c40bb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62A-0-0, dag_ret:0, dag_status:2, start_time:1727337635655630, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.656659] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=4] task finish process(ret=0, start_time=1727337635656184, end_time=1727337635656658, runtime=474, *this={this:0x7f9264c40bb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62A-0-0, dag_ret:0, dag_status:2, start_time:1727337635655630, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.656646] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9265b01330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:492}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:12, rowkey_column_count:11, schema_rowkey_col_cnt:9, full_stored_col_cnt:59, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753437040, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=28 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.656669] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=18] succeed to build merge ctx(tablet_id={id:492}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753437040, storage_schema:{this:0x7f9265b01330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:492}, table_count:1, [{i:0, table_key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8624c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635655540, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.10ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.656740] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C627-0-0, dag_ret:0, dag_status:2, start_time:1727337635654771, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265b014c0, output_block_cnt_arr:0x7f9265b014c8, concurrent_cnt:1, estimate_row_cnt:1362, estimate_occupy_size:2097152, latest_update_ts:1727337635655540, estimated_finish_time:1727337652656740}) [2024-09-26 16:00:35.656746] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c3fc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C627-0-0, dag_ret:0, dag_status:2, start_time:1727337635654771, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.656750] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=4] task finish process(ret=0, start_time=1727337635655539, end_time=1727337635656750, runtime=1211, *this={this:0x7f9264c3fc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C627-0-0, dag_ret:0, dag_status:2, start_time:1727337635654771, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.656897] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337635623735, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.656907] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=2] schedule one task(task=0x7f926652a3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.656907] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.656915] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=5] schedule one task(task=0x7f9264ae8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.656918] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=3] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.656923] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635656921, task_id:YB42C0A8002D-0006230056D8C62D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.656926] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=3] schedule one task(task=0x7f9264c41b40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.656926] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=16] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.656979] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=342, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:21, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:23}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b0c060, count:6, allocator:0x7f92865a27d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:23, datum_buffer:{capacity:32, datums:0x7f9265b0c390, local_datums:0x7f9265b0c390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,]}) [2024-09-26 16:00:35.656982] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=492, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:57, schema_rowkey_cnt:9, rowkey_cnt:11, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:9, column_cnt:59}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265a7e060, count:11, allocator:0x7f92bc9047d8, capacity:11, init_cnt:11}, datum_utils:{is_oracle_mode:false, rowkey_cnt:11, col_cnt:11, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:59, datum_buffer:{capacity:59, datums:0x7f9287dd6050, local_datums:0x7f9265a7e440}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,col_id=43:NOP,col_id=44:NOP,col_id=45:NOP,col_id=46:NOP,col_id=47:NOP,col_id=48:NOP,col_id=49:NOP,col_id=50:NOP,col_id=51:NOP,col_id=52:NOP,col_id=53:NOP,col_id=54:NOP,col_id=55:NOP,col_id=56:NOP,col_id=57:NOP,col_id=58:NOP,]}) [2024-09-26 16:00:35.657014] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.657048] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.657050] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.657059] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337635623676, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.657077] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635656935, end_time=1727337635657077, runtime=142, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62A-0-0, dag_ret:0, dag_status:2, start_time:1727337635655630, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.657085] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1455, dag_cnt=27, dag_cnts_[dag.get_type()]=27, &dag=0x7f9264c407f0, dag={ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635655630, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.657092] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62A-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635655618, task_id:YB42C0A8002D-0006230056D8C62A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.657127] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.657130] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.657132] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:35.657135] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.657138] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.657144] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.657195] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=19][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.657233] ERROR issue_dba_error (ob_log.cpp:1875) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.657236] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.657244] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=0] partition merge iter row count(i=0, row_count=61, ghost_row_count=0, pkey={tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337635623618, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.657272] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635657270, task_id:YB42C0A8002D-0006230056D8C62E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.657277] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=4] schedule one task(task=0x7f9264c42070, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.657264] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635656938, end_time=1727337635657264, runtime=326, *this={this:0x7f9264ae8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C627-0-0, dag_ret:0, dag_status:2, start_time:1727337635654771, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.657292] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2521, dag_cnt=26, dag_cnts_[dag.get_type()]=26, &dag=0x7f9264c3f860, dag={ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C627-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635654771, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.657301] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C627-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337635654767, task_id:YB42C0A8002D-0006230056D8C627-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.657329] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:35.657341] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337635657340, task_id:YB42C0A8002D-0006230056D8C62F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.657344] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=3] schedule one task(task=0x7f9264c425a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.657372] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=455, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:36, schema_rowkey_cnt:7, rowkey_cnt:9, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:7, column_cnt:38}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b58060, count:9, allocator:0x7f928642a7d8, capacity:9, init_cnt:9}, datum_utils:{is_oracle_mode:false, rowkey_cnt:9, col_cnt:9, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:38, datum_buffer:{capacity:38, datums:0x7f9263b59340, local_datums:0x7f9263b583f0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,]}) [2024-09-26 16:00:35.657397] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=19][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.657400] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.657402] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:35.657406] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.657408] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864df20, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:503, tablet_addr:[814042-5586-0-2305843643796283392-2](offset=1171456,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:377}, data_tablet_id:{id:377}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4164021307, row_count:17}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864e5fc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844760487772160-0](offset=1163264,size=1631,type=2,seq=0), type:"MAJOR", tablet_id:{id:377}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563080818000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843989541144159-0](offset=1164895,size=913,type=2,seq=0), type:"MINI", tablet_id:{id:377}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557779816000, upper_trans_version:1726121557779816000}, {i:1, addr:[814042-5586-0-2305843969140050416-0](offset=1165808,size=894,type=2,seq=0), type:"MINI", tablet_id:{id:377}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986451466000, upper_trans_version:1726149986451466000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337635623795, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdb5b0, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220773832}, this:0x7f92a3bdb5b0, timestamp:1727334220773832, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:80, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293334630000, v:0}, rec_scn:{val:1726264800181268000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513780224-2](offset=1167360,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843176718586294-0](offset=1167798,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880514130-0](offset=1167954,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447034164-0](offset=1168180,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7e310, 0x7f92a3bdb5b0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.657464] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.657470] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.657617] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337635623754, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.657626] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=18][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.657627] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.657639] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.657642] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.657647] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101010}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:35.657657] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.657660] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.657669] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=0] partition merge iter row count(i=0, row_count=140, ghost_row_count=0, pkey={tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337635623598, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.657663] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101010}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101010}, table_count:1, [{i:0, table_key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8642d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635657122, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=523us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.657696] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635657330, end_time=1727337635657696, runtime=366, *this={this:0x7f926652a3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C626-0-0, dag_ret:0, dag_status:2, start_time:1727337635654763, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.657705] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2942, dag_cnt=25, dag_cnts_[dag.get_type()]=25, &dag=0x7f9264c3f330, dag={ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C626-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635654763, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.657707] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=37] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62D-0-0, dag_ret:0, dag_status:2, start_time:1727337635656926, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3470, output_block_cnt_arr:0x7f92728f3478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337635657122, estimated_finish_time:1727337652657707}) [2024-09-26 16:00:35.657713] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C626-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635654757, task_id:YB42C0A8002D-0006230056D8C626-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.657714] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c41b40, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62D-0-0, dag_ret:0, dag_status:2, start_time:1727337635656926, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.657719] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=5] task finish process(ret=0, start_time=1727337635657121, end_time=1727337635657719, runtime=598, *this={this:0x7f9264c41b40, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62D-0-0, dag_ret:0, dag_status:2, start_time:1727337635656926, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.657741] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=2] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.657749] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337635657748, task_id:YB42C0A8002D-0006230056D8C630-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.657753] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=3] schedule one task(task=0x7f9264c42ad0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.657754] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=10] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337635623795, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.657752] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337635623714, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.657763] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9264d5d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=41, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.657772] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.657769] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=17] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.657775] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.657775] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.657779] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.657780] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.657781] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:35.657783] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.657786] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.657780] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264d5d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:377}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:43, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.657788] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:140}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.657797] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:377}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9264d5d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:377}, table_count:1, [{i:0, table_key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635657391, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=387us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.657815] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62F-0-0, dag_ret:0, dag_status:2, start_time:1727337635657344, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264d5d480, output_block_cnt_arr:0x7f9264d5d488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337635657391, estimated_finish_time:1727337652657814}) [2024-09-26 16:00:35.657804] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:140}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:140}, table_count:1, [{i:0, table_key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8622d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635656429, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.36ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.657820] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c425a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62F-0-0, dag_ret:0, dag_status:2, start_time:1727337635657344, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.657824] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=4] task finish process(ret=0, start_time=1727337635657390, end_time=1727337635657824, runtime=434, *this={this:0x7f9264c425a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62F-0-0, dag_ret:0, dag_status:2, start_time:1727337635657344, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.657789] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864eec0, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[814042-5586-0-2305843650238771200-2](offset=1208320,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:297}, data_tablet_id:{id:297}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726063409221604000, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2361333062, row_count:24}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844531780800512-0](offset=1200128,size=1418,type=2,seq=0), type:"MAJOR", tablet_id:{id:297}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063202518238000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843919747962250-0](offset=1201546,size=848,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726063409221604000, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557514739000, upper_trans_version:1726121557514739000}, {i:1, addr:[814042-5586-0-2305843920821704922-0](offset=1202394,size=849,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149603136610000, upper_trans_version:1726149603136610000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337635623816, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd86d0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220478269}, this:0x7f92a3bd86d0, timestamp:1727334220478269, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:74, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726264804632655000, v:0}, rec_scn:{val:1726264804632655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513817088-2](offset=1204224,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843183161074102-0](offset=1204662,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880551000-0](offset=1204824,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447071034-0](offset=1205050,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7eae0, 0x7f92a3bd86d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.657826] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62B-0-0, dag_ret:0, dag_status:2, start_time:1727337635656194, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f460, output_block_cnt_arr:0x7f9264b1f468, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337635656429, estimated_finish_time:1727337652657826}) [2024-09-26 16:00:35.657840] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=50] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.657833] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c410e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62B-0-0, dag_ret:0, dag_status:2, start_time:1727337635656194, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.657847] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.657848] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=15] task finish process(ret=0, start_time=1727337635656426, end_time=1727337635657848, runtime=1422, *this={this:0x7f9264c410e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62B-0-0, dag_ret:0, dag_status:2, start_time:1727337635656194, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.657858] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=2] schedule one task(task=0x7f927282a3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.657862] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=3] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.657880] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.657883] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.657885] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:35.657889] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.657892] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.657898] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.657907] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.657906] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=140, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:3, allocator:0x7f92c4a5e7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:35.657912] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101009}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:35.657927] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:101009}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101009}, table_count:1, [{i:0, table_key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8641da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635656534, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.38ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.657946] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62C-0-0, dag_ret:0, dag_status:2, start_time:1727337635656202, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b21470, output_block_cnt_arr:0x7f9264b21478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337635656534, estimated_finish_time:1727337652657946}) [2024-09-26 16:00:35.657951] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=4] succeed to init merge ctx(task={this:0x7f9264c41610, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62C-0-0, dag_ret:0, dag_status:2, start_time:1727337635656202, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.657955] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=4] task finish process(ret=0, start_time=1727337635656532, end_time=1727337635657955, runtime=1423, *this={this:0x7f9264c41610, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62C-0-0, dag_ret:0, dag_status:2, start_time:1727337635656202, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.657959] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.658016] ERROR issue_dba_error (ob_log.cpp:1875) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=13][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.658019] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.658029] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337635623714, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.658058] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635657876, end_time=1727337635658058, runtime=182, *this={this:0x7f927282a3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62B-0-0, dag_ret:0, dag_status:2, start_time:1727337635656194, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.658070] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1875, dag_cnt=24, dag_cnts_[dag.get_type()]=24, &dag=0x7f9264c40d20, dag={ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635656194, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.658078] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635656188, task_id:YB42C0A8002D-0006230056D8C62B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.658135] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=377, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:41, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:43}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b0060, count:5, allocator:0x7f92c244a7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:43, datum_buffer:{capacity:43, datums:0x7f92684b12b0, local_datums:0x7f92684b0360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,]}) [2024-09-26 16:00:35.658173] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.658203] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.658193] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337635623773, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.658205] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.658207] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752949376, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=20, max_schema_version_in_memtable=1710828752949376, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.658211] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=0] partition merge iter row count(i=0, row_count=6, ghost_row_count=0, pkey={tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337635623795, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.658226] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.658229] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.658228] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635658105, end_time=1727337635658228, runtime=123, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62F-0-0, dag_ret:0, dag_status:2, start_time:1727337635657344, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.658235] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=891, dag_cnt=23, dag_cnts_[dag.get_type()]=23, &dag=0x7f9264c421e0, dag={ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635657344, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.658241] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62F-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635657340, task_id:YB42C0A8002D-0006230056D8C62F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.658234] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:323}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:22, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752949376, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.658251] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:323}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752949376, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:323}, table_count:1, [{i:0, table_key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635657875, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=357us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.658269] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337635623816, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.658278] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=16, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.658289] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.658292] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.658291] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=26] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62E-0-0, dag_ret:0, dag_status:2, start_time:1727337635657277, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3480, output_block_cnt_arr:0x7f9287db3488, concurrent_cnt:1, estimate_row_cnt:205, estimate_occupy_size:0, latest_update_ts:1727337635657875, estimated_finish_time:1727337652658290}) [2024-09-26 16:00:35.658297] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c42070, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62E-0-0, dag_ret:0, dag_status:2, start_time:1727337635657277, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.658304] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=7] task finish process(ret=0, start_time=1727337635657866, end_time=1727337635658304, runtime=438, *this={this:0x7f9264c42070, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62E-0-0, dag_ret:0, dag_status:2, start_time:1727337635657277, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.658297] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:297}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:18, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.658316] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:297}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:297}, table_count:1, [{i:0, table_key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635657770, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=525us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.658351] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=29] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C630-0-0, dag_ret:0, dag_status:2, start_time:1727337635657753, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1490, output_block_cnt_arr:0x7f92728f1498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337635657770, estimated_finish_time:1727337652658351}) [2024-09-26 16:00:35.658357] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c42ad0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C630-0-0, dag_ret:0, dag_status:2, start_time:1727337635657753, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.658362] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=5] task finish process(ret=0, start_time=1727337635657768, end_time=1727337635658362, runtime=594, *this={this:0x7f9264c42ad0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C630-0-0, dag_ret:0, dag_status:2, start_time:1727337635657753, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.658378] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=1] schedule one task(task=0x7f9264c49260, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.658384] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=3] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.658387] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=2] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.658391] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337635658389, task_id:YB42C0A8002D-0006230056D8C631-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.658403] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=11] schedule one task(task=0x7f9264c43000, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.658408] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635658407, task_id:YB42C0A8002D-0006230056D8C632-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.658411] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=2] schedule one task(task=0x7f9264c43530, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.658463] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=323, range=0x7f92ab056200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:20, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:22}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:5, allocator:0x7f92636447d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,]}) [2024-09-26 16:00:35.658538] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.658542] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.658544] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:35.658549] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.658560] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=297, range=0x7f92aa6d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:16, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:18}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665ac060, count:6, allocator:0x7f92bc9047d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:18, datum_buffer:{capacity:32, datums:0x7f92665ac390, local_datums:0x7f92665ac390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,]}) [2024-09-26 16:00:35.658587] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.658553] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706339704000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864bfe0, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:499, tablet_addr:[809012-1445-0-2305843639500640256-2](offset=495616,size=587,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101065}, data_tablet_id:{id:332}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706339704000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2355617684, row_count:171}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864c6e0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844381456211968-0](offset=466944,size=1278,type=2,seq=0), type:"MAJOR", tablet_id:{id:101065}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843926189679870-0](offset=468222,size=854,type=2,seq=0), type:"MINI", tablet_id:{id:101065}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337635623834, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdc550, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220895690}, this:0x7f92a3bdc550, timestamp:1727334220895690, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:82, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265274283037001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696450560-2](offset=471040,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843183160340812-0](offset=471372,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879817710-0](offset=471534,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7db40, 0x7f92a3bdc550, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.658592] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=39] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.658598] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.658629] ERROR issue_dba_error (ob_log.cpp:1875) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.658632] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.658640] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337635623816, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.658647] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101009, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a5e060, count:4, allocator:0x7f92636847d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a5e350, local_datums:0x7f9264a5e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:35.658658] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635658503, end_time=1727337635658658, runtime=155, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C630-0-0, dag_ret:0, dag_status:2, start_time:1727337635657753, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.658667] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=913, dag_cnt=22, dag_cnts_[dag.get_type()]=22, &dag=0x7f9264c42710, dag={ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C630-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635657753, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.658673] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C630-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337635657748, task_id:YB42C0A8002D-0006230056D8C630-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.658683] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.658721] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635658719, task_id:YB42C0A8002D-0006230056D8C633-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.658725] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=3] schedule one task(task=0x7f9264c43a60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.658739] ERROR issue_dba_error (ob_log.cpp:1875) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.658745] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.658743] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.658749] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.658752] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:35.658757] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.658754] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337635623735, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.658762] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.658770] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.658777] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635658619, end_time=1727337635658777, runtime=158, *this={this:0x7f9264c49260, type:1, status:2, dag:{ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62C-0-0, dag_ret:0, dag_status:2, start_time:1727337635656202, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.658788] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2586, dag_cnt=21, dag_cnts_[dag.get_type()]=21, &dag=0x7f9264c41250, dag={ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635656202, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.658796] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62C-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635656198, task_id:YB42C0A8002D-0006230056D8C62C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.658823] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=45][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.658861] ERROR issue_dba_error (ob_log.cpp:1875) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.658861] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337635658859, task_id:YB42C0A8002D-0006230056D8C634-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.658865] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=4] schedule one task(task=0x7f9264c43f90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.658864] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.658874] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=1] partition merge iter row count(i=0, row_count=171, ghost_row_count=0, pkey={tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337635623773, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.658896] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.658899] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.658900] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:35.658895] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635658398, end_time=1727337635658895, runtime=497, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62E-0-0, dag_ret:0, dag_status:2, start_time:1727337635657277, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.658904] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.658905] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1628, dag_cnt=20, dag_cnts_[dag.get_type()]=20, &dag=0x7f9264c41cb0, dag={ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635657277, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.658912] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62E-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635657270, task_id:YB42C0A8002D-0006230056D8C62E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.658907] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706339704027, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8654c80, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:522, tablet_addr:[809012-1445-0-2305843664196722688-2](offset=516096,size=610,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:334}, data_tablet_id:{id:334}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706339704027, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2773161616, row_count:13383}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86553c8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844791625629696-0](offset=507904,size=1660,type=2,seq=0), type:"MAJOR", tablet_id:{id:334}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843977729328764-0](offset=509564,size=902,type=2,seq=0), type:"MINI", tablet_id:{id:334}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337635623883, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bddcc0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221095878}, this:0x7f92a3bddcc0, timestamp:1727334221095878, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:85, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265286142368001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843367843975168-2](offset=512000,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843196045283662-0](offset=512334,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879858684-0](offset=512508,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843215372636894-0](offset=512734,size=192,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc57f0, 0x7f92a3bddcc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.658955] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=48] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.658961] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.658961] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.658964] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.658966] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:35.658970] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.658973] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.659006] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635659004, task_id:YB42C0A8002D-0006230056D8C635-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.659011] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=4] schedule one task(task=0x7f9264c444c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.659025] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.659048] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.659050] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.659052] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:35.659055] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.659090] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337635623834, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.659106] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92665a9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.659120] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.659124] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.659129] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92665a9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101065}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.659057] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077691154184001, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8650e00, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:497, tablet_addr:[809012-1445-0-2305843637353218048-2](offset=557056,size=585,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:451}, data_tablet_id:{id:451}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077691154184001, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2119849913, row_count:31}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86514b8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844404004851712-0](offset=528384,size=1299,type=2,seq=0), type:"MAJOR", tablet_id:{id:451}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563069147000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843980950574355-0](offset=529683,size=905,type=2,seq=0), type:"MINI", tablet_id:{id:451}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986382094000, upper_trans_version:1726149986382094000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337635623912, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdfc00, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334222657108}, this:0x7f92a3bdfc00, timestamp:1727334222657108, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753331112, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:89, clock:641728512}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293246986000, v:0}, rec_scn:{val:1726265293246986000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696516096-2](offset=536576,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843170275504460-0](offset=536908,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879883234-0](offset=537058,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446403268-0](offset=537284,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc4080, 0x7f92a3bdfc00, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.659147] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=90] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.659128] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101010, range=0x7f92a9952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ee060, count:4, allocator:0x7f92b32387d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92664ee350, local_datums:0x7f92664ee350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:35.659153] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.659148] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101065}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92665a9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101065}, table_count:1, [{i:0, table_key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635658532, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=595us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.659183] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C631-0-0, dag_ret:0, dag_status:2, start_time:1727337635658402, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665a9490, output_block_cnt_arr:0x7f92665a9498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337635658532, estimated_finish_time:1727337652659183}) [2024-09-26 16:00:35.659190] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=25][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.659191] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264c43000, type:15, status:2, dag:{ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C631-0-0, dag_ret:0, dag_status:2, start_time:1727337635658402, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.659197] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=5] task finish process(ret=0, start_time=1727337635658530, end_time=1727337635659197, runtime=667, *this={this:0x7f9264c43000, type:15, status:2, dag:{ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C631-0-0, dag_ret:0, dag_status:2, start_time:1727337635658402, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.659235] ERROR issue_dba_error (ob_log.cpp:1875) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.659248] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=13][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.659258] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337635623754, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.659277] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635659092, end_time=1727337635659276, runtime=184, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62D-0-0, dag_ret:0, dag_status:2, start_time:1727337635656926, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.659286] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2360, dag_cnt=19, dag_cnts_[dag.get_type()]=19, &dag=0x7f9264c41780, dag={ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C62D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635656926, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.659294] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C62D-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635656921, task_id:YB42C0A8002D-0006230056D8C62D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.659322] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337635623883, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.659323] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337635623850, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.659331] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92665fd330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=19, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.659333] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92665ff330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.659353] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.659356] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.659370] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.659373] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.659362] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92665ff330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:333}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.659378] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92665fd330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:334}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:21, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.659398] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:334}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92665fd330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:334}, table_count:1, [{i:0, table_key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8654c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635658878, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=498us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.659381] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:333}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92665ff330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:333}, table_count:1, [{i:0, table_key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8651da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635658956, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=403us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.659449] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=54] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C632-0-0, dag_ret:0, dag_status:2, start_time:1727337635658411, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665ff490, output_block_cnt_arr:0x7f92665ff498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337635658956, estimated_finish_time:1727337652659449}) [2024-09-26 16:00:35.659456] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c43530, type:15, status:2, dag:{ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C632-0-0, dag_ret:0, dag_status:2, start_time:1727337635658411, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.659461] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=5] task finish process(ret=0, start_time=1727337635658954, end_time=1727337635659461, runtime=507, *this={this:0x7f9264c43530, type:15, status:2, dag:{ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C632-0-0, dag_ret:0, dag_status:2, start_time:1727337635658411, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.659464] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=31] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C634-0-0, dag_ret:0, dag_status:2, start_time:1727337635658865, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665fd4a0, output_block_cnt_arr:0x7f92665fd4a8, concurrent_cnt:1, estimate_row_cnt:993, estimate_occupy_size:0, latest_update_ts:1727337635658878, estimated_finish_time:1727337652659464}) [2024-09-26 16:00:35.659471] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c43f90, type:15, status:2, dag:{ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C634-0-0, dag_ret:0, dag_status:2, start_time:1727337635658865, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.659476] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=4] task finish process(ret=0, start_time=1727337635658876, end_time=1727337635659475, runtime=599, *this={this:0x7f9264c43f90, type:15, status:2, dag:{ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C634-0-0, dag_ret:0, dag_status:2, start_time:1727337635658865, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.659493] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337635623865, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.659502] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.659524] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.659527] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.659533] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101066}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.659549] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101066}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101066}, table_count:1, [{i:0, table_key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8653ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635658740, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=791us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.659573] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C633-0-0, dag_ret:0, dag_status:2, start_time:1727337635658725, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665a7490, output_block_cnt_arr:0x7f92665a7498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337635658740, estimated_finish_time:1727337652659573}) [2024-09-26 16:00:35.659579] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c43a60, type:15, status:2, dag:{ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C633-0-0, dag_ret:0, dag_status:2, start_time:1727337635658725, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.659583] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=4] task finish process(ret=0, start_time=1727337635658737, end_time=1727337635659583, runtime=846, *this={this:0x7f9264c43a60, type:15, status:2, dag:{ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C633-0-0, dag_ret:0, dag_status:2, start_time:1727337635658725, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.659614] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=3] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.659621] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=4] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.659624] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=2] schedule one task(task=0x7f92685b23c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.659626] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=2] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.659630] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635659629, task_id:YB42C0A8002D-0006230056D8C636-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.659634] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=4] schedule one task(task=0x7f9264c449f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.659670] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=334, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:19, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:21}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685ea060, count:8, allocator:0x7f92636287d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:21, datum_buffer:{capacity:32, datums:0x7f92685ea3e0, local_datums:0x7f92685ea3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,]}) [2024-09-26 16:00:35.659712] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101066, range=0x7f92aad52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:7}, cols_desc:{ObIArray:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:7, allocator:0x7f92b33d87d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9263a383b0, local_datums:0x7f9263a383b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:35.659864] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=35][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.659891] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=21][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.659902] ERROR issue_dba_error (ob_log.cpp:1875) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.659905] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.659915] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=0] partition merge iter row count(i=0, row_count=182, ghost_row_count=0, pkey={tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337635623865, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.659925] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.659928] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.659936] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635659674, end_time=1727337635659935, runtime=261, *this={this:0x7f92685b23c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C633-0-0, dag_ret:0, dag_status:2, start_time:1727337635658725, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.659937] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=0] partition merge iter row count(i=0, row_count=153, ghost_row_count=0, pkey={tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337635623883, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.659945] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1220, dag_cnt=18, dag_cnts_[dag.get_type()]=18, &dag=0x7f9264c436a0, dag={ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C633-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635658725, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.659952] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C633-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635658719, task_id:YB42C0A8002D-0006230056D8C633-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.659955] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635659644, end_time=1727337635659955, runtime=311, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C634-0-0, dag_ret:0, dag_status:2, start_time:1727337635658865, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.660008] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101065, range=0x7f92aaad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b58060, count:6, allocator:0x7f92c244a7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9263b58390, local_datums:0x7f9263b58390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:35.660031] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337635660029, task_id:YB42C0A8002D-0006230056D8C637-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.660036] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=4] schedule one task(task=0x7f9264c44f20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.660039] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.660043] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1178, dag_cnt=17, dag_cnts_[dag.get_type()]=17, &dag=0x7f9264c43bd0, dag={ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C634-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635658865, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.660051] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C634-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337635658859, task_id:YB42C0A8002D-0006230056D8C634-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.660053] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=14][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.660056] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.660058] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:35.660062] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.660064] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.660066] ERROR issue_dba_error (ob_log.cpp:1875) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.660068] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635660067, task_id:YB42C0A8002D-0006230056D8C638-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.660070] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241548806527000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.660071] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=2] schedule one task(task=0x7f9264c45450, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.660069] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.660075] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=0] partition merge iter row count(i=0, row_count=15, ghost_row_count=0, pkey={tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337635623834, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.660087] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.660090] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.660092] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:35.660091] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635659980, end_time=1727337635660091, runtime=111, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C631-0-0, dag_ret:0, dag_status:2, start_time:1727337635658402, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.660096] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.660099] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.660099] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1697, dag_cnt=16, dag_cnts_[dag.get_type()]=16, &dag=0x7f9264c42c40, dag={ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C631-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635658402, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.660107] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.660107] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C631-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337635658389, task_id:YB42C0A8002D-0006230056D8C631-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.660141] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635660139, task_id:YB42C0A8002D-0006230056D8C639-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.660145] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=3] schedule one task(task=0x7f9264c45980, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.660151] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.660155] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.660157] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:35.660161] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.660165] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.660167] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.660169] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:35.660172] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.660174] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.660178] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.660164] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864cf80, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[814042-5586-0-2305843644870074368-2](offset=1220608,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:378}, data_tablet_id:{id:378}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2166189977, row_count:350}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864d65c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844352465928192-0](offset=1212416,size=1251,type=2,seq=0), type:"MAJOR", tablet_id:{id:378}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563080818000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843914379265251-0](offset=1213667,size=843,type=2,seq=0), type:"MINI", tablet_id:{id:378}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557779816000, upper_trans_version:1726121557779816000}, {i:1, addr:[814042-5586-0-2305843914379266094-0](offset=1214510,size=843,type=2,seq=0), type:"MINI", tablet_id:{id:378}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986451466000, upper_trans_version:1726149986451466000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337635623929, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdf430, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221743022}, this:0x7f92a3bdf430, timestamp:1727334221743022, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:88, clock:641728512}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293334630000, v:0}, rec_scn:{val:1726265293334630000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513829376-2](offset=1216512,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843176718635446-0](offset=1216950,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880563282-0](offset=1217106,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447083316-0](offset=1217332,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7f2b0, 0x7f92a3bdf430, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.660203] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=39] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.660210] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.660238] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337635623912, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.660251] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=14, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.660261] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.660264] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.660259] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=333, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:7, allocator:0x7f92636627d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9263a383b0, local_datums:0x7f9263a383b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:35.660269] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:451}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:16, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.660287] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:451}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:451}, table_count:1, [{i:0, table_key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8650e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635659045, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.22ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.660322] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=30] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C635-0-0, dag_ret:0, dag_status:2, start_time:1727337635659010, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515470, output_block_cnt_arr:0x7f9268515478, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337635659045, estimated_finish_time:1727337652660322}) [2024-09-26 16:00:35.660329] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c444c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C635-0-0, dag_ret:0, dag_status:2, start_time:1727337635659010, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.660333] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=4] task finish process(ret=0, start_time=1727337635659043, end_time=1727337635660333, runtime=1290, *this={this:0x7f9264c444c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C635-0-0, dag_ret:0, dag_status:2, start_time:1727337635659010, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.660420] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.660457] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.660472] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.660482] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=0] partition merge iter row count(i=0, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337635623850, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.660502] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635660234, end_time=1727337635660502, runtime=268, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C632-0-0, dag_ret:0, dag_status:2, start_time:1727337635658411, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.660511] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2100, dag_cnt=15, dag_cnts_[dag.get_type()]=15, &dag=0x7f9264c43170, dag={ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C632-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635658411, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.660518] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C632-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635658407, task_id:YB42C0A8002D-0006230056D8C632-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.660544] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752511928, storage_schema:{this:0x7f9263a3f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.660553] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.660556] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.660560] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a3f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:121}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752511928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.660574] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:121}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f9263a3f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635660048, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=510us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.660614] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C637-0-0, dag_ret:0, dag_status:2, start_time:1727337635660035, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f9263a3f480, output_block_cnt_arr:0x7f9263a3f488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337635660048, estimated_finish_time:1727337652660614}) [2024-09-26 16:00:35.660620] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c44f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C637-0-0, dag_ret:0, dag_status:2, start_time:1727337635660035, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:35.660625] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=4] task finish process(ret=0, start_time=1727337635660046, end_time=1727337635660624, runtime=578, *this={this:0x7f9264c44f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C637-0-0, dag_ret:0, dag_status:2, start_time:1727337635660035, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:35.662526] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337635624061, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.662535] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337635623929, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.662541] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=14] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753355744, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753355744, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.662557] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=12] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.662561] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.662551] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.662567] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:456}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753355744, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.662580] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=24] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.662584] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.662591] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:378}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.662587] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:456}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753355744, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:456}, table_count:1, [{i:0, table_key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8634c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635660161, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.40ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.662613] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C639-0-0, dag_ret:0, dag_status:2, start_time:1727337635660144, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a7490, output_block_cnt_arr:0x7f92a23a7498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337635660161, estimated_finish_time:1727337652662612}) [2024-09-26 16:00:35.662629] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=16] succeed to init merge ctx(task={this:0x7f9264c45980, type:15, status:2, dag:{ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C639-0-0, dag_ret:0, dag_status:2, start_time:1727337635660144, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.662610] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:378}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:378}, table_count:1, [{i:0, table_key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635660146, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.44ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.662635] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=5] task finish process(ret=0, start_time=1727337635660157, end_time=1727337635662635, runtime=2478, *this={this:0x7f9264c45980, type:15, status:2, dag:{ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C639-0-0, dag_ret:0, dag_status:2, start_time:1727337635660144, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.662642] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=26] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C636-0-0, dag_ret:0, dag_status:2, start_time:1727337635659634, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab480, output_block_cnt_arr:0x7f92684ab488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337635660146, estimated_finish_time:1727337652662642}) [2024-09-26 16:00:35.662649] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c449f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C636-0-0, dag_ret:0, dag_status:2, start_time:1727337635659634, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.662655] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=5] task finish process(ret=0, start_time=1727337635660143, end_time=1727337635662655, runtime=2512, *this={this:0x7f9264c449f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C636-0-0, dag_ret:0, dag_status:2, start_time:1727337635659634, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.662657] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337635624043, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.662668] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752725472, storage_schema:{this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752725472, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.662677] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:35.662677] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.662680] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.662681] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=2][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.662684] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:215}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752725472, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.662700] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:215}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752725472, storage_schema:{this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:215}, table_count:1, [{i:0, table_key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635660082, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.60ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.662717] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C638-0-0, dag_ret:0, dag_status:2, start_time:1727337635660070, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665a7460, output_block_cnt_arr:0x7f92665a7468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337635660082, estimated_finish_time:1727337652662717}) [2024-09-26 16:00:35.662722] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c45450, type:15, status:2, dag:{ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C638-0-0, dag_ret:0, dag_status:2, start_time:1727337635660070, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.662726] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=4] task finish process(ret=0, start_time=1727337635660080, end_time=1727337635662726, runtime=2646, *this={this:0x7f9264c45450, type:15, status:2, dag:{ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C638-0-0, dag_ret:0, dag_status:2, start_time:1727337635660070, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.662738] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=4] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.662745] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=3] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.662748] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.662750] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=1] schedule one task(task=0x7f9268486030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.662754] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=1] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.662759] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635662757, task_id:YB42C0A8002D-0006230056D8C63A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.662762] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=3] schedule one task(task=0x7f9264c45eb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.662818] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=456, range=0x7f92aaed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d34060, count:7, allocator:0x7f92bc9127d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f9287d343b0, local_datums:0x7f9287d343b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:35.662826] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=378, range=0x7f92aab52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d38060, count:5, allocator:0x7f9269ffe7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9287d38360, local_datums:0x7f9287d38360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:35.662884] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.662899] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=121, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dda060, count:5, allocator:0x7f92636287d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9287dda360, local_datums:0x7f9287dda360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:35.662921] ERROR issue_dba_error (ob_log.cpp:1875) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.662924] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.662919] INFO [STORAGE] dump_info (ob_sstable_struct.cpp:262) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=15] dump merge info(msg="macro block builder close", output_row_per_s=0, new_macro_KB_per_s=0, *this={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:57, merge_start_time:1727337635662861, merge_finish_time:1727337635662918, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:0, max:0, sum:0, count:0}}, {type:"cost_time", info:{min:0, max:0, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:0, max:0, sum:0, count:0}}, {type:"incremental_row_count", info:{min:0, max:0, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:0, snapshot_version:0, start_scn:0, end_scn:0}, macro_id_list:"", comment:""}) [2024-09-26 16:00:35.662933] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=0] partition merge iter row count(i=0, row_count=45, ghost_row_count=0, pkey={tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337635624061, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.662934] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=14] partition merge iter row count(i=0, row_count=0, ghost_row_count=0, pkey={tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, table={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337635623946, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.662942] ERROR issue_dba_error (ob_log.cpp:1875) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.662946] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.662947] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1544) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=10] merge macro blocks ok(idx_=0, task={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C637-0-0, dag_ret:0, dag_status:2, start_time:1727337635660035, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:35.662955] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635662773, end_time=1727337635662954, runtime=181, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C639-0-0, dag_ret:0, dag_status:2, start_time:1727337635660144, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.662960] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=8] task finish process(ret=0, start_time=1727337635662858, end_time=1727337635662960, runtime=102, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C637-0-0, dag_ret:0, dag_status:2, start_time:1727337635660035, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:35.662965] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2821, dag_cnt=14, dag_cnts_[dag.get_type()]=14, &dag=0x7f9264c455c0, dag={ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C639-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635660144, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.662973] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C639-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635660139, task_id:YB42C0A8002D-0006230056D8C639-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.662956] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337635623929, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.662994] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635662772, end_time=1727337635662994, runtime=222, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C636-0-0, dag_ret:0, dag_status:2, start_time:1727337635659634, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.663006] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3372, dag_cnt=13, dag_cnts_[dag.get_type()]=13, &dag=0x7f9264c44630, dag={ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C636-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635659634, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.663015] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C636-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337635659629, task_id:YB42C0A8002D-0006230056D8C636-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.663020] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=451, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:14, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:16}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fc060, count:4, allocator:0x7f92636607d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:16, datum_buffer:{capacity:32, datums:0x7f92665fc350, local_datums:0x7f92665fc350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,]}) [2024-09-26 16:00:35.663051] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.663055] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.663057] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:35.663061] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.663076] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.663082] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.663085] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.663089] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=7][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.663088] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635663085, task_id:YB42C0A8002D-0006230056D8C63B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.663100] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=11] schedule one task(task=0x7f9264c463e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.663108] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337635663106, task_id:YB42C0A8002D-0006230056D8C63C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.663110] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=2] schedule one task(task=0x7f9264c46910, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.663100] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337635623912, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.663130] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635662993, end_time=1727337635663130, runtime=137, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C635-0-0, dag_ret:0, dag_status:2, start_time:1727337635659010, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.663139] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4128, dag_cnt=12, dag_cnts_[dag.get_type()]=12, &dag=0x7f9264c44100, dag={ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C635-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635659010, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.663146] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C635-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635659004, task_id:YB42C0A8002D-0006230056D8C635-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.663153] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.663156] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.663158] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:35.663178] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=17] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.663201] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=215, range=0x7f92ab152200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:6}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dd4060, count:3, allocator:0x7f92b32387d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9287dd4320, local_datums:0x7f9287dd4320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:35.663183] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b863eec0, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:499, tablet_addr:[816017-1849-0-2305843639502065664-2](offset=1921024,size=587,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101016}, data_tablet_id:{id:154}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726074677203981000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:398898191, row_count:578}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844419038609408-0](offset=1900544,size=1313,type=2,seq=0), type:"MAJOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726074015354550000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844024975361313-0](offset=1901857,size=946,type=2,seq=0), type:"MINOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:1726074677203981000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726205802467242000, upper_trans_version:1726205802467242000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337635624172, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd5fc0, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209508808}, this:0x7f92a3bd5fc0, timestamp:1727334209508808, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:69, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277803107894000, v:0}, rec_scn:{val:1726241804194470000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771625984-2](offset=1904640,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161774413-0](offset=1904973,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881251311-0](offset=1905135,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc7f00, 0x7f92a3bd5fc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.663224] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=40] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.663230] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.663253] ERROR issue_dba_error (ob_log.cpp:1875) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.663256] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635663254, task_id:YB42C0A8002D-0006230056D8C63D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.663255] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.663259] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=3] schedule one task(task=0x7f9264c46e40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.663263] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=0] partition merge iter row count(i=0, row_count=9, ghost_row_count=0, pkey={tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337635624043, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.663275] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.663278] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.663279] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:35.663278] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635663174, end_time=1727337635663277, runtime=103, *this={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C638-0-0, dag_ret:0, dag_status:2, start_time:1727337635660070, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.663282] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.663285] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3215, dag_cnt=11, dag_cnts_[dag.get_type()]=11, &dag=0x7f9264c45090, dag={ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C638-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635660070, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.663291] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C638-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337635660067, task_id:YB42C0A8002D-0006230056D8C638-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.663325] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.663330] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.663333] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:35.663337] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.663340] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.663285] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b864fe60, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:510, tablet_addr:[816017-1849-0-2305843651313397760-2](offset=2093056,size=598,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101063}, data_tablet_id:{id:297}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726063409221604000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1864622635, row_count:24}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86505a8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844568288894976-0](offset=2072576,size=1452,type=2,seq=0), type:"MAJOR", tablet_id:{id:101063}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063202518238000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843928338769324-0](offset=2074028,size=856,type=2,seq=0), type:"MINI", tablet_id:{id:101063}, scn_range:{start_scn:{val:1726063409221604000, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557514739000, upper_trans_version:1726121557514739000}, {i:1, addr:[816017-1849-0-2305843928338770180-0](offset=2074884,size=856,type=2,seq=0), type:"MINI", tablet_id:{id:101063}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149603136610000, upper_trans_version:1726149603136610000}, {i:2, addr:[816017-1849-0-2305843927265029212-0](offset=2075740,size=855,type=2,seq=0), type:"MINI", tablet_id:{id:101063}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726207962418314000, upper_trans_version:1726207962418314000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337635624219, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd8ea0, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220519573}, this:0x7f92a3bd8ea0, timestamp:1727334220519573, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:75, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726264804632655000, v:0}, rec_scn:{val:1726264804632655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843593331331072-2](offset=2084864,size=544,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046856736-0](offset=2085408,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881431758-0](offset=2085582,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bca610, 0x7f92a3bd8ea0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.663349] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.663356] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.663357] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.663413] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337635624078, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.663423] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.663505] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=63] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.663510] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.663516] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:154}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.663535] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:154}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:154}, table_count:1, [{i:0, table_key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8623ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635663045, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=469us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.663560] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63A-0-0, dag_ret:0, dag_status:2, start_time:1727337635662762, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a7480, output_block_cnt_arr:0x7f92a23a7488, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337635663045, estimated_finish_time:1727337652663559}) [2024-09-26 16:00:35.663555] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337635624097, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.663567] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c45eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63A-0-0, dag_ret:0, dag_status:2, start_time:1727337635662762, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.663572] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=5] task finish process(ret=0, start_time=1727337635663044, end_time=1727337635663572, runtime=528, *this={this:0x7f9264c45eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63A-0-0, dag_ret:0, dag_status:2, start_time:1727337635662762, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.663568] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.663582] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.663585] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.663598] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101015}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.663623] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=20] succeed to build merge ctx(tablet_id={id:101015}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101015}, table_count:1, [{i:0, table_key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635663318, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=270us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.663649] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63B-0-0, dag_ret:0, dag_status:2, start_time:1727337635663100, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3490, output_block_cnt_arr:0x7f92728f3498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337635663318, estimated_finish_time:1727337652663649}) [2024-09-26 16:00:35.663655] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c463e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63B-0-0, dag_ret:0, dag_status:2, start_time:1727337635663100, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.663661] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=5] task finish process(ret=0, start_time=1727337635663317, end_time=1727337635663660, runtime=343, *this={this:0x7f9264c463e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63B-0-0, dag_ret:0, dag_status:2, start_time:1727337635663100, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.664439] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337635624172, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.664455] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.664493] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=33] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.664497] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.664503] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101016}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.664523] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101016}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101016}, table_count:1, [{i:0, table_key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635663147, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.35ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.664553] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63C-0-0, dag_ret:0, dag_status:2, start_time:1727337635663110, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1490, output_block_cnt_arr:0x7f92728f1498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337635663147, estimated_finish_time:1727337652664553}) [2024-09-26 16:00:35.664587] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=33] succeed to init merge ctx(task={this:0x7f9264c46910, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63C-0-0, dag_ret:0, dag_status:2, start_time:1727337635663110, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.664593] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=6] task finish process(ret=0, start_time=1727337635663145, end_time=1727337635664593, runtime=1448, *this={this:0x7f9264c46910, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63C-0-0, dag_ret:0, dag_status:2, start_time:1727337635663110, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.664618] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=3] schedule one task(task=0x7f9264c80250, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.664624] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=3] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.664629] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=2] schedule one task(task=0x7f9264d76030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.664631] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=1] schedule one task(task=0x7f92658c2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.664637] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635664635, task_id:YB42C0A8002D-0006230056D8C63E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.664641] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=4] schedule one task(task=0x7f9264c47370, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.664639] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337635624219, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.664661] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=22] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.664675] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.664679] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.664684] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101063}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.664707] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=18] succeed to build merge ctx(tablet_id={id:101063}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101063}, table_count:1, [{i:0, table_key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635663271, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.41ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.664723] INFO [STORAGE.COMPACTION] get_merged_sstable (ob_tablet_merge_task.cpp:1285) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=3] create new merged sstable(ctx.param_.tablet_id_={id:121}, snapshot_version=1726241132764173999, ctx.param_.merge_type_=3, ctx.create_snapshot_version_=0, table_mode_flag=0) [2024-09-26 16:00:35.664731] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63D-0-0, dag_ret:0, dag_status:2, start_time:1727337635663259, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685154a0, output_block_cnt_arr:0x7f92685154a8, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337635663271, estimated_finish_time:1727337652664731}) [2024-09-26 16:00:35.664733] INFO [STORAGE] close (ob_index_block_builder.cpp:815) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=5] succeed to close sstable index builder(res={root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_block_ids_.count():0, other_block_ids_.count():0, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, data_column_cnt:10, data_column_checksums:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], row_count:0, max_merged_trans_version:0, contain_uncommitted_row:false, occupy_size:0, original_size:0, data_checksum:0, use_old_macro_block_count:0, compressor_type:1, root_row_store_type:0, nested_offset:0, nested_size:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}) [2024-09-26 16:00:35.664738] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c46e40, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63D-0-0, dag_ret:0, dag_status:2, start_time:1727337635663259, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.664743] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=4] task finish process(ret=0, start_time=1727337635663270, end_time=1727337635664742, runtime=1472, *this={this:0x7f9264c46e40, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63D-0-0, dag_ret:0, dag_status:2, start_time:1727337635663259, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.664747] INFO [STORAGE] init (ob_sstable.cpp:118) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=8] succeeded to init sstable(ret=0, this={ObITable:{this:0x7f9263a3e128, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9263a3e128, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:null, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:35.664767] INFO [STORAGE.COMPACTION] create_sstable (ob_tablet_merge_ctx.cpp:367) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=19] succeed to merge sstable(param={table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_logic_seq:0, schema_version:1710828752511928, create_snapshot_version:0, progressive_merge_round:0, progressive_merge_step:0, is_ready_for_read:true, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, index_type:0, root_block_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), root_block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, root_row_store_type:0, latest_row_store_type:1, data_index_tree_height:0, data_block_macro_meta_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), data_block_macro_meta:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, use_old_macro_block_count:0, row_count:0, rowkey_column_cnt:5, column_cnt:10, column_checksums:[], data_checksum:0, occupy_size:0, original_size:0, max_merged_trans_version:1726241132764173999, ddl_scn:{val:0, v:0}, contain_uncommitted_row:false, is_meta_root:false, compressor_type:1, encrypt_id:0, master_key_id:0, recycle_version:0, nested_offset:0, nested_size:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}, table_key={tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_merge_info={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:2870, merge_start_time:1727337635660048, merge_finish_time:1727337635662918, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"cost_time", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"incremental_row_count", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:1, snapshot_version:0, start_scn:1726211867305630000, end_scn:1726241548806527000}, macro_id_list:"", comment:""}) [2024-09-26 16:00:35.664793] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101015, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:6, allocator:0x7f92636847d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:35.664845] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101016, range=0x7f92aae56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d58060, count:6, allocator:0x7f926368c7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9264d58390, local_datums:0x7f9264d58390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:35.664902] ERROR issue_dba_error (ob_log.cpp:1875) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.664909] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=27][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.664906] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.664916] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337635624097, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.664938] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635664764, end_time=1727337635664938, runtime=174, *this={this:0x7f9264d76030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63B-0-0, dag_ret:0, dag_status:2, start_time:1727337635663100, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.664948] ERROR issue_dba_error (ob_log.cpp:1875) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.664948] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1847, dag_cnt=10, dag_cnts_[dag.get_type()]=10, &dag=0x7f9264c46020, dag={ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635663100, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.664951] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.664956] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635663085, task_id:YB42C0A8002D-0006230056D8C63B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.664961] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337635624172, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.664983] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635664813, end_time=1727337635664983, runtime=170, *this={this:0x7f92658c2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63C-0-0, dag_ret:0, dag_status:2, start_time:1727337635663110, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.664989] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.664992] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.664994] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:35.664993] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1882, dag_cnt=9, dag_cnts_[dag.get_type()]=9, &dag=0x7f9264c46550, dag={ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635663110, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.664998] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.665000] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63C-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635663106, task_id:YB42C0A8002D-0006230056D8C63C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.665002] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.665009] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.665074] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=154, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92728fa060, count:5, allocator:0x7f92636827d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92728fa360, local_datums:0x7f92728fa360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:35.665269] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=2] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.665280] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337635665278, task_id:YB42C0A8002D-0006230056D8C63F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.665284] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=4] schedule one task(task=0x7f9264c478a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.665289] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337635665288, task_id:YB42C0A8002D-0006230056D8C640-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.665292] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=2] schedule one task(task=0x7f9264c47dd0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.665315] ERROR issue_dba_error (ob_log.cpp:1875) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.665318] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.665329] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337635624078, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.665350] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635665048, end_time=1727337635665350, runtime=302, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63A-0-0, dag_ret:0, dag_status:2, start_time:1727337635662762, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.665371] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2608, dag_cnt=8, dag_cnts_[dag.get_type()]=8, &dag=0x7f9264c45af0, dag={ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635662762, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.665378] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63A-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635662757, task_id:YB42C0A8002D-0006230056D8C63A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.665377] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=14] multi version scan iterator init succ(param.table_id_=101063, range=0x7f92aac56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dda060, count:8, allocator:0x7f92636827d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287dda3e0, local_datums:0x7f9287dda3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:35.665406] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.665423] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.665426] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.665429] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:35.665433] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.665454] ERROR issue_dba_error (ob_log.cpp:1875) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=14][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.665473] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=19][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.665482] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=0] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337635624219, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.665437] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706139416032, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f926e072bc0, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[812168-2208-0-2305843644869693440-2](offset=839680,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:328}, data_tablet_id:{id:328}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706139416032, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:867269498, row_count:786}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f926e07329c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[812168-2208-0-2305844589762490368-0](offset=831488,size=1472,type=2,seq=0), type:"MAJOR", tablet_id:{id:328}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562828712000}}, minor_tables:{{i:0, addr:[812168-2208-0-2305843935853721024-0](offset=832960,size=863,type=2,seq=0), type:"MINI", tablet_id:{id:328}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986144852000, upper_trans_version:1726149986144852000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337635624299, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bda610, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220621826}, this:0x7f92a3bda610, timestamp:1727334220621826, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:78, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265257450254001, v:0}, rec_scn:{val:1726264907253071001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[812168-2208-0-2305843366770556928-2](offset=835584,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[812168-2208-0-2305843176718254413-0](offset=835917,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843251880182249-0](offset=836073,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843216446702283-0](offset=836299,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc8ea0, 0x7f92a3bda610, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.665501] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=64] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.665508] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.665503] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635665343, end_time=1727337635665503, runtime=160, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63D-0-0, dag_ret:0, dag_status:2, start_time:1727337635663259, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.665513] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2254, dag_cnt=7, dag_cnts_[dag.get_type()]=7, &dag=0x7f9264c46a80, dag={ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635663259, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.665521] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63D-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337635663254, task_id:YB42C0A8002D-0006230056D8C63D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.665527] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337635624259, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.665535] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=15, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.665555] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.665559] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.665561] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:35.665560] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=22] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.665564] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.665565] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.665569] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:452}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:17, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.665586] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:452}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:452}, table_count:1, [{i:0, table_key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8652d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635664984, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=583us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.665569] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706539141067, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f926e107fe0, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:2, version:2, length:516, tablet_addr:[809012-1445-0-2305843657754198016-2](offset=442368,size=604,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:330}, data_tablet_id:{id:330}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706539141067, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3803410633, row_count:15330}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f926e108704, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844604794474496-0](offset=430080,size=1486,type=2,seq=0), type:"MAJOR", tablet_id:{id:330}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562805980000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843948738221518-0](offset=431566,size=875,type=2,seq=0), type:"MINI", tablet_id:{id:330}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986144852000, upper_trans_version:1726149986144852000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337635624358, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdade0, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220705914}, this:0x7f92a3bdade0, timestamp:1727334220705914, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:79, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265269081598001, v:0}, rec_scn:{val:1726264907253071001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696413696-2](offset=434176,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843189602754892-0](offset=434508,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879780852-0](offset=434676,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446300886-0](offset=434902,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc9670, 0x7f92a3bdade0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:35.665608] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=38] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.665608] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63E-0-0, dag_ret:0, dag_status:2, start_time:1727337635664641, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f480, output_block_cnt_arr:0x7f9264b1f488, concurrent_cnt:1, estimate_row_cnt:10, estimate_occupy_size:0, latest_update_ts:1727337635664984, estimated_finish_time:1727337652665607}) [2024-09-26 16:00:35.665614] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.665614] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c47370, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63E-0-0, dag_ret:0, dag_status:2, start_time:1727337635664641, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.665619] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=4] task finish process(ret=0, start_time=1727337635664982, end_time=1727337635665618, runtime=636, *this={this:0x7f9264c47370, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63E-0-0, dag_ret:0, dag_status:2, start_time:1727337635664641, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.665655] INFO [STORAGE] build_new_table_store (ob_tablet_table_store.cpp:1206) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=21] succeed to build new table store(major_tables_={cnt:1, is_inited:true}, minor_tables_={cnt:3, is_inited:true}, memtables_={count:1}, PRINT_TS(*this)={ObTabletTableStore_Pretty:this:0x7f92aa751ca8, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, memtables:{count:1}, is_ready_for_read:true, table_array:{ [2870] [ YB42C0A8002D-0006230056D8C637-0-0 ] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [2870] [ YB42C0A8002D-0006230056D8C637-0-0 ] MAJOR MAJOR 1725120003420849000 1725120003420849000 {val:0, v:0} {val:1726077600415496000, v:0} 0 false [2870] [ YB42C0A8002D-0006230056D8C637-0-0 ] MINOR MINOR 1726178585269925002 1726178585269925002 {val:1726077259159067000, v:0} {val:1726178585269925002, v:0} 0 false [2870] [ YB42C0A8002D-0006230056D8C637-0-0 ] MINI 1726211867106157000 1726211867106157000 {val:1726178585269925002, v:0} {val:1726211867305630000, v:0} 0 false [2870] [ YB42C0A8002D-0006230056D8C637-0-0 ] MINI 1726241132764173999 1726241132764173999 {val:1726211867305630000, v:0} {val:1726241548806527000, v:0} 0 false [2870] [ YB42C0A8002D-0006230056D8C637-0-0 ] MEM ACTIVE 9223372036854775807 18446744073709547609 {val:1726241548806527000, v:0} {val:4611686018427387903, v:0} 3 unused }}) [2024-09-26 16:00:35.665747] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=2] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.665756] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337635665754, task_id:YB42C0A8002D-0006230056D8C641-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.665760] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=3] schedule one task(task=0x7f9264c48300, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.665765] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337635665764, task_id:YB42C0A8002D-0006230056D8C642-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.665767] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=2] schedule one task(task=0x7f9264c48830, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.665810] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.665816] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.665819] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:35.665823] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.665826] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.665845] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.665836] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=452, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:15, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:17}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b58060, count:5, allocator:0x7f92864107d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:17, datum_buffer:{capacity:32, datums:0x7f9263b58360, local_datums:0x7f9263b58360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,]}) [2024-09-26 16:00:35.665864] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337635624299, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.665886] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=21] update column cnt(column_cnt_=26, store_column_cnt_=26, column_cnt_=26, column_array_.count()=0) [2024-09-26 16:00:35.665890] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=4] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92a23a7480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=26, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.665901] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.665908] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.665911] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.665920] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=0] partition merge iter row count(i=0, row_count=10, ghost_row_count=0, pkey={tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337635624259, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.665939] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635665808, end_time=1727337635665939, runtime=131, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63E-0-0, dag_ret:0, dag_status:2, start_time:1727337635664641, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.665949] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1308, dag_cnt=6, dag_cnts_[dag.get_type()]=6, &dag=0x7f9264c46fb0, dag={ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635664641, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.665956] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63E-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635664635, task_id:YB42C0A8002D-0006230056D8C63E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.665979] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.665982] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.665984] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:35.665987] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.665990] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.665995] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.666003] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=50] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.666009] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:328}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:28, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.666064] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=49] succeed to build merge ctx(tablet_id={id:328}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92a23a7480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:328}, table_count:1, [{i:0, table_key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e072bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635665418, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=589us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.666094] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63F-0-0, dag_ret:0, dag_status:2, start_time:1727337635665284, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a75d0, output_block_cnt_arr:0x7f92a23a75d8, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337635665418, estimated_finish_time:1727337652666093}) [2024-09-26 16:00:35.666101] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c478a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63F-0-0, dag_ret:0, dag_status:2, start_time:1727337635665284, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.666105] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=4] task finish process(ret=0, start_time=1727337635665416, end_time=1727337635666105, runtime=689, *this={this:0x7f9264c478a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63F-0-0, dag_ret:0, dag_status:2, start_time:1727337635665284, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.666242] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=2] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.666253] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] succeed to add sys task(task={start_time:1727337635666252, task_id:YB42C0A8002D-0006230056D8C643-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.666257] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=4] schedule one task(task=0x7f9264c48d60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.666324] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=328, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:26, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:28}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dd2060, count:5, allocator:0x7f92865a27d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:28, datum_buffer:{capacity:32, datums:0x7f9287dd2360, local_datums:0x7f9287dd2360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,]}) [2024-09-26 16:00:35.666361] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=18][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.666390] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337635624469, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.666401] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=10] update column cnt(column_cnt_=29, store_column_cnt_=29, column_cnt_=29, column_array_.count()=0) [2024-09-26 16:00:35.666403] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92728f3490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.666408] ERROR issue_dba_error (ob_log.cpp:1875) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=16][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.666415] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.666411] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.666418] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.666420] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337635624299, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.666423] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:329}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.666451] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635666293, end_time=1727337635666451, runtime=158, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63F-0-0, dag_ret:0, dag_status:2, start_time:1727337635665284, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.666461] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1177, dag_cnt=5, dag_cnts_[dag.get_type()]=5, &dag=0x7f9264c474e0, dag={ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C63F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635665284, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.666449] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=21] succeed to build merge ctx(tablet_id={id:329}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92728f3490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:329}, table_count:1, [{i:0, table_key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e089f20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635665967, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=454us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.666484] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C63F-0-0] [lt=21] succeed to del sys task(removed_task={start_time:1727337635665278, task_id:YB42C0A8002D-0006230056D8C63F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.666493] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=36] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C641-0-0, dag_ret:0, dag_status:2, start_time:1727337635665759, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f35f0, output_block_cnt_arr:0x7f92728f35f8, concurrent_cnt:1, estimate_row_cnt:62, estimate_occupy_size:0, latest_update_ts:1727337635665967, estimated_finish_time:1727337652666493}) [2024-09-26 16:00:35.666499] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c48300, type:15, status:2, dag:{ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C641-0-0, dag_ret:0, dag_status:2, start_time:1727337635665759, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.666503] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=4] task finish process(ret=0, start_time=1727337635665965, end_time=1727337635666503, runtime=538, *this={this:0x7f9264c48300, type:15, status:2, dag:{ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C641-0-0, dag_ret:0, dag_status:2, start_time:1727337635665759, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.666512] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=2] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.666509] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337635624358, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.666519] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=9] update column cnt(column_cnt_=12, store_column_cnt_=12, column_cnt_=12, column_array_.count()=0) [2024-09-26 16:00:35.666533] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337635624513, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.666533] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=14] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9264b21490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.666542] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=25, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.666552] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.666554] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.666556] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=12] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.666560] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.666558] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:332}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:27, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.666565] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b21490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:330}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.666570] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=329, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:6, allocator:0x7f9275d787d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9263b5e390, local_datums:0x7f9263b5e390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:35.666575] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:332}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:332}, table_count:1, [{i:0, table_key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8648b00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635665806, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=751us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.666592] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C642-0-0, dag_ret:0, dag_status:2, start_time:1727337635665767, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a45490, output_block_cnt_arr:0x7f9263a45498, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337635665806, estimated_finish_time:1727337652666592}) [2024-09-26 16:00:35.666597] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c48830, type:15, status:2, dag:{ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C642-0-0, dag_ret:0, dag_status:2, start_time:1727337635665767, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.666583] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:330}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9264b21490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:330}, table_count:1, [{i:0, table_key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e107fe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635665549, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.01ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.666601] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=4] task finish process(ret=0, start_time=1727337635665804, end_time=1727337635666601, runtime=797, *this={this:0x7f9264c48830, type:15, status:2, dag:{ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C642-0-0, dag_ret:0, dag_status:2, start_time:1727337635665767, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.666614] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C640-0-0, dag_ret:0, dag_status:2, start_time:1727337635665292, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b215f0, output_block_cnt_arr:0x7f9264b215f8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337635665549, estimated_finish_time:1727337652666613}) [2024-09-26 16:00:35.666632] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=17] succeed to init merge ctx(task={this:0x7f9264c47dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C640-0-0, dag_ret:0, dag_status:2, start_time:1727337635665292, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.666638] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=6] task finish process(ret=0, start_time=1727337635665543, end_time=1727337635666638, runtime=1095, *this={this:0x7f9264c47dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C640-0-0, dag_ret:0, dag_status:2, start_time:1727337635665292, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.666662] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.666665] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:35.666667] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:35.666670] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:35.666674] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:35.666681] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.666706] INFO [MDS] init_for_evict_medium_info (ob_tablet_mds_data.cpp:414) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=13] succeeded to init mds data(ret=0, finish_medium_scn=1726077600415496000, src_medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, src_extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}) [2024-09-26 16:00:35.666730] ERROR issue_dba_error (ob_log.cpp:1875) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.666735] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.666745] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=0] partition merge iter row count(i=0, row_count=47, ghost_row_count=0, pkey={tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337635624469, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.666767] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635666539, end_time=1727337635666767, runtime=228, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C641-0-0, dag_ret:0, dag_status:2, start_time:1727337635665759, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.666778] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1019, dag_cnt=4, dag_cnts_[dag.get_type()]=4, &dag=0x7f9264c47f40, dag={ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C641-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635665759, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.666722] INFO [STORAGE] init_for_merge (ob_tablet.cpp:363) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=5] succeeded to init tablet for mini/minor/major merge(ret=0, param={sstable:0x7f9263a3e128, snapshot_version:1726241132764173999, clog_checkpoint_scn:{val:1726241548806527000, v:0}, multi_version_start:1726178585269925002, need_report:false, storage_schema:{this:0x7f9263a3f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, rebuild_seq:0, update_with_major_flag:false, need_check_sstable:false, ddl_info:{keep_old_ddl_sstable:true, ddl_start_scn:{val:0, v:0}, ddl_commit_scn:{val:0, v:0}, ddl_checkpoint_scn:{val:0, v:0}, ddl_snapshot_version:0, ddl_execution_id:-1, data_format_version:0}, allow_duplicate_sstable:false, merge_type:"MINI_MERGE", need_check_transfer_seq:true, transfer_seq:0}, old_tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337635623130000, hold_ref_cnt:true, ref_cnt:3, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337635623946, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}, this={this:0x7f9263a36060, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x7f92aa752450, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726241548806527000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726241132764173999, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f9263a36670, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}, {i:2, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726241132764173999, upper_trans_version:1726241132764173999}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x7f9263a36520, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9263a37880, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9263a37950, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{redefined:false, snapshot_version:0, schema_version:0, data_tablet_id:{id:0}, hidden_tablet_id:{id:0}, lob_meta_tablet_id:{id:50121}, lob_piece_tablet_id:{id:60121}, is_old_mds:true}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:35.666797] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C641-0-0] [lt=16] succeed to del sys task(removed_task={start_time:1727337635665754, task_id:YB42C0A8002D-0006230056D8C641-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.666824] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=5] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.666828] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=3] schedule one task(task=0x7f9265918030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.666814] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=78] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f92728f0060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}}, ref_cnt:0, upper_trans_version:1725120003420849000, timestamp:0}, this:0x7f92728f0060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1725120003420849000, max_merged_trans_version:1725120003420849000, data_macro_block_count:1, nested_size:4096, nested_offset:589824, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:84, row_count_:14, occupy_size_:1604, original_size_:1914, data_checksum_:3587602083, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:1, data_micro_block_count_:1, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:2, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1725120003420849000, max_merged_trans_version_:1725120003420849000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f92728f0310, column_checksum_count:10, data_root_info:{addr:[0-0-0-3458764933653594112-0](offset=0,size=391,type=3,seq=0), block_data:{buf:0x7f92728f0360, size:391, extra_buf:0x7f92728f04e7, extra_size:424, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-3458765009889263616-0](offset=0,size=462,type=3,seq=0), block_data:{buf:0x7f92728f068f, size:462, extra_buf:null, extra_size:0, type:1}}, data_block_count:1, other_block_count:0, linked_block_count:0, data_block_ids:0x7f92728f085d, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:true, nested_offset:589824, nested_size:4096}}, valid_for_reading:true}) [2024-09-26 16:00:35.666852] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=35] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f92728f0060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}}, ref_cnt:0, upper_trans_version:1726178585269925002, timestamp:0}, this:0x7f92728f0060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726178585269925002, max_merged_trans_version:1726178585269925002, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726178585269925002, v:0}, meta:{basic_meta:{version:1, length:94, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726178585269925002, max_merged_trans_version_:1726178585269925002, recycle_version:1726077600415496000, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726178585269925002, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:1, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f92728f0310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:35.666864] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=11] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f92728f0060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}}, ref_cnt:0, upper_trans_version:1726211867106157000, timestamp:0}, this:0x7f92728f0060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726211867106157000, max_merged_trans_version:1726211867106157000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726211867305630000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726211867106157000, max_merged_trans_version_:1726211867106157000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726211867305630000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f92728f0310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:35.666870] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=330, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:7, allocator:0x7f92a33547d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f92684b43b0, local_datums:0x7f92684b43b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:35.666876] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=11] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f92728f0060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f92728f0060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f92728f0310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:35.666918] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.666921] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf9de5b1 0xf9dcfd6 0xe51c6fb 0xe519c03 0xe518f89 0xe516f8c 0xf01158d 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.666939] INFO [STORAGE] reset (ob_tablet.cpp:159) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=0] reset tablet(this=0x7f9263a36060, ls_id={id:1}, tablet_id={id:121}, lbt()="0x121f1f9c 0xe52cc09 0xe52c6c5 0xe52bd13 0x4fd348f 0xf01110c 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:35.666946] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1250) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=0][errcode=-4184] sstable merge finish(ret=-4184, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f9263a3f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C637-0-0, dag_ret:0, dag_status:2, start_time:1727337635660035, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f9263a3f480, output_block_cnt_arr:0x7f9263a3f488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337635660048, estimated_finish_time:1727337652660614}, compaction_filter:NULL, time_guard:total=4.74ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, task={this:0x7f9264c80250, type:16, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C637-0-0, dag_ret:0, dag_status:2, start_time:1727337635660035, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:35.666972] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635664720, end_time=1727337635666972, runtime=2252, *this={this:0x7f9264c80250, type:16, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C637-0-0, dag_ret:0, dag_status:2, start_time:1727337635660035, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:35.666979] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=6944, dag_cnt=3, dag_cnts_[dag.get_type()]=3, &dag=0x7f9264c44b60, dag={ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C637-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635660035, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}) [2024-09-26 16:00:35.666985] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C637-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635660029, task_id:YB42C0A8002D-0006230056D8C637-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.667039] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=332, range=0x7f92a9ed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:25, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:27}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d36060, count:6, allocator:0x7f9275d787d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:27, datum_buffer:{capacity:32, datums:0x7f9287d36390, local_datums:0x7f9287d36390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,]}) [2024-09-26 16:00:35.667053] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.667057] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.667072] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:35.667066] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=0] partition merge iter row count(i=0, row_count=141, ghost_row_count=0, pkey={tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337635624358, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.667086] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337635666839, end_time=1727337635667086, runtime=247, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C640-0-0, dag_ret:0, dag_status:2, start_time:1727337635665292, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.667095] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1803, dag_cnt=2, dag_cnts_[dag.get_type()]=2, &dag=0x7f9264c47a10, dag={ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C640-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635665292, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.667100] ERROR issue_dba_error (ob_log.cpp:1875) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.667102] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C640-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635665288, task_id:YB42C0A8002D-0006230056D8C640-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.667102] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.667110] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337635624513, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.667128] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337635667012, end_time=1727337635667128, runtime=116, *this={this:0x7f9265918030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C642-0-0, dag_ret:0, dag_status:2, start_time:1727337635665767, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.667136] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1369, dag_cnt=1, dag_cnts_[dag.get_type()]=1, &dag=0x7f9264c48470, dag={ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C642-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635665767, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.667142] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C642-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337635665764, task_id:YB42C0A8002D-0006230056D8C642-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.667178] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337635624558, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:35.667190] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:35.667209] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:35.667212] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:35.667217] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101067}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:35.667235] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101067}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101067}, table_count:1, [{i:0, table_key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b86a2d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337635666657, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=558us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:35.667273] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C643-0-0, dag_ret:0, dag_status:2, start_time:1727337635666257, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f4a0, output_block_cnt_arr:0x7f9264b1f4a8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337635666657, estimated_finish_time:1727337652667272}) [2024-09-26 16:00:35.667279] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c48d60, type:15, status:2, dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C643-0-0, dag_ret:0, dag_status:2, start_time:1727337635666257, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.667284] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=5] task finish process(ret=0, start_time=1727337635666655, end_time=1727337635667284, runtime=629, *this={this:0x7f9264c48d60, type:15, status:2, dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C643-0-0, dag_ret:0, dag_status:2, start_time:1727337635666257, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.667923] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=2] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:35.667973] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101067, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b58060, count:8, allocator:0x7f92636607d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b583e0, local_datums:0x7f9263b583e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:35.668137] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:35.668171] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:35.668173] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:35.668181] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:35.668183] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:35.668185] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:8, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a220, hex: CF18F6C100220600, int: 1726236509739215},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a0a8, hex: 0000000000000000, int: 0},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a0e0, hex: 3501000000000000, int: 309},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a118, hex: 3501000000000000, int: 309},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a150, hex: 1600000000000000, int: 22},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a188, hex: 0800000000000000, int: 8},idx=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a1c0, hex: 909E2A1D0A2D0BE8, int: -1726236510704198000},idx=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a1f8, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:5017, column_count:8, rowkey_column_count:8, has_column_checksum:0, row_count:227, row_store_type:0, opt:5, var_column_count:0, row_offset:13522, original_length:14370, max_merged_trans_version:1726236510704198000, data_length:14370, data_zlength:14370, data_checksum:3929059508, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f926fe04090, buf_size:14370, data_size:14370, row_count:227, column_count:8, max_merged_trans_version:1726236510704198000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1778324432, row_count_delta:227, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:14370}) [2024-09-26 16:00:35.668199] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=14][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:35.668201] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=2][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:00:35.668203] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=2][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:00:35.668204] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=1][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:00:35.668206] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=2][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:00:35.668208] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=1] partition merge iter row count(i=0, row_count=227, ghost_row_count=0, pkey={tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337635624558, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:35.668221] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=11][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:35.668229] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=3][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:35.668232] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:35.668234] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337635667944, end_time=1727337635668233, runtime=289, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C643-0-0, dag_ret:0, dag_status:2, start_time:1727337635666257, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:35.668240] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C643-0-0, dag_ret:0, dag_status:2, start_time:1727337635666257, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:35.668247] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=1990, dag_cnt=0, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264c489a0, dag={ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C643-0-0, dag_ret:-4184, dag_status:5, start_time:1727337635666257, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:35.668255] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C643-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337635666252, task_id:YB42C0A8002D-0006230056D8C643-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:35.674982] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:35.674994] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337635674978) [2024-09-26 16:00:35.674999] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337635574826, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:35.675024] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223364910956, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.675030] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:35.675037] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:35.675042] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:35.675045] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:35.675051] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.675053] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.675055] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337635675002) [2024-09-26 16:00:35.675071] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=19] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:35.680771] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337635304381]}) [2024-09-26 16:00:35.692959] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:35.692969] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.723096] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29] success to notify_need_writing_throttling True [2024-09-26 16:00:35.723106] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.745787] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=4] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:35.746876] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337635746875, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007164, add_bytes: 0 [2024-09-26 16:00:35.753521] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:35.753530] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.775302] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:35.775322] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337635775298) [2024-09-26 16:00:35.775328] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337635675002, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:35.775347] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223365011762, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.775354] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:35.775363] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:35.775369] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:35.775372] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:35.775381] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.775383] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.775385] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337635775331) [2024-09-26 16:00:35.780871] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337635304381]}) [2024-09-26 16:00:35.783799] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:35.783807] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.787572] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:35.787703] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=25, wash_time=1) [2024-09-26 16:00:35.788324] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=408941, clean_num=31457) [2024-09-26 16:00:35.805406] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:35.805417] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337635805403], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:35.805425] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337635805403], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:35.805429] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:35.805435] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:35.814042] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:35.814057] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.821890] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.821891] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.821904] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.821907] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.821913] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.821919] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.821926] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.821932] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.821938] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.821942] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.821946] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:35.844088] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:35.844098] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.863881] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337636793290, now=[mts=1727337634863519]) [2024-09-26 16:00:35.863890] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337636793290, now=[mts=1727337634863519], now0=[mts=1727337634863519], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:35.863896] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a3116d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500714, assoc_session_id:3221500714, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337634862784, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:35.863915] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22f01f8, id:3221500714, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3116d80}) [2024-09-26 16:00:35.863923] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500714, *tx_desc={this:0x7f92a3116d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500714, assoc_session_id:3221500714, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337634862784, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:35.863930] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=3221500714, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3116d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500714, assoc_session_id:3221500714, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337634862784, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337634863495, use_das=false, nested_level=0, session={this:0x7f92a22f01f8, id:3221500714, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3116d80}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:35.863948] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:35.863952] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:35.863957] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:35.863960] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:35.863962] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:35.863966] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:35.863970] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=999320, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337636863290) [2024-09-26 16:00:35.863973] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:35.863980] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:35.863982] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:35.863984] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:35.863989] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:35.864011] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2169) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2] need check schema(new_schema_version=1, cached_tenant_schema_version=1, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = ? ORDER BY row_id, column_name) [2024-09-26 16:00:35.864033] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:35.864035] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:35.874635] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:35.874643] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.875603] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:35.875615] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337635875599) [2024-09-26 16:00:35.875621] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337635775331, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:35.875626] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:35.875634] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:35.875637] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:35.875965] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885746-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:35.875999] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885746-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:35.876032] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337635876032, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=402, wlock_time=8, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:35.876042] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:35.876058] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223365112509, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.876064] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:35.876070] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:35.876076] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:35.876078] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:35.876084] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.876086] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.876088] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337635876047) [2024-09-26 16:00:35.881482] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337635805403]}) [2024-09-26 16:00:35.889876] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=4] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-09-26 16:00:35.889887] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=8] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=687) [2024-09-26 16:00:35.905058] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:35.905067] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.912747] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:134, tid:2892}]) [2024-09-26 16:00:35.935227] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:35.935234] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.935250] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:35.945918] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:35.965466] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:35.965473] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:35.972297] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:35.975961] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:35.975968] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:35.975990] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337635975959) [2024-09-26 16:00:35.975995] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337635875624, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:35.976011] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223365211753, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:35.976018] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:35.976024] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:35.976029] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:35.976031] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:35.976037] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.976041] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:35.976043] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337635975999) [2024-09-26 16:00:35.982067] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337635805403]}) [2024-09-26 16:00:35.989009] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:35.989128] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=51, refresh_score_time=30, wash_time=2) [2024-09-26 16:00:35.989739] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=440398, clean_num=31457) [2024-09-26 16:00:35.995793] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:35.995801] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.000796] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500720, support_ssl=false) [2024-09-26 16:00:36.000829] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221500720, sess.client_addr_="127.0.0.1:36332") [2024-09-26 16:00:36.000835] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500720, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:36.000855] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:36.000863] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=2] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:36.000909] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9D-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:36.001143] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9D-0-0] [lt=5][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:36.001151] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9D-0-0] [lt=7][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"bEr2={tV0-2R&fP<||zp"}, ret=-4043) [2024-09-26 16:00:36.001158] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9D-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:36.001161] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9D-0-0] [lt=2][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:36.001168] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9D-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:36.001190] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9D-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500720, proxy_sessid:0}) [2024-09-26 16:00:36.001195] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9D-0-0] [lt=4] mark session id unused(sessid=3221500720) [2024-09-26 16:00:36.001198] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9D-0-0] [lt=1][errcode=0] server close connection(sessid=3221500720, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:36.001243] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9D-0-0] [lt=45][errcode=-4018] get session fail(ret=-4018, sessid=3221500720, proxy_sessid=0) [2024-09-26 16:00:36.001247] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9D-0-0] [lt=3][errcode=-4016] session is null [2024-09-26 16:00:36.001250] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9D-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500720, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:36.001298] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500720, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337636000614, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:36.001310] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221500720, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:36.001313] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=2] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500720, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337636000614, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:36.001317] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500720, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:36.020239] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185900-0-0] [lt=46][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:36.020255] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185900-0-0] [lt=14][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:36.020264] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185900-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:36.020280] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185900-0-0] [lt=15][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:36.026066] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.026077] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.030885] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: SHARED_BLOCK_ReaderWriter_IO, mode: read, size: 4096.00, iops: 52.56, rt: 2547.28) [2024-09-26 16:00:36.030896] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:36.030917] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 9397.00) [2024-09-26 16:00:36.030920] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:36.030926] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:36.030929] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:36.039020] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:36.039035] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=14][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:36.039040] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:36.039045] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:36.039049] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:36.039052] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:36.039055] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:36.039058] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:36.039060] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:36.039062] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:36.039064] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:36.039066] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:36.039087] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:36.039091] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:36.039098] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:36.039102] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:36.039105] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:36.039107] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:36.039110] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:36.039112] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:36.039115] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:36.039121] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:36.039128] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:36.039130] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:36.039132] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:36.039137] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:36.039141] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587D-0-0] [lt=4][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:36.039144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587D-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:36.039147] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:36.039149] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:36.039151] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:36.039154] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337636038943, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:36.039158] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587D-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:36.039161] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587D-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:36.039187] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587D-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:36.039192] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8587D-0-0] [lt=4][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:36.049463] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=9] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:36.049765] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:36.049775] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:36.049779] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:36.049795] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:36.049799] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:36.049801] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:36.049810] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:36.049812] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:36.049814] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:36.049816] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:36.049818] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:36.049820] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:36.049822] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:36.049824] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:36.049830] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:36.049832] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:36.049835] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:36.049838] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:36.049840] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:36.049842] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:36.049845] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:36.049850] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:36.049855] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:36.049859] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:36.049861] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:36.049866] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:36.049869] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E60-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:36.049872] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2582][RSAsyncTask3][T0][YB42C0A8002D-0006230057684E60-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:36.049875] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:36.049878] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:36.049880] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:36.049885] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337636049719, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:36.049888] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:36.049933] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2582][RSAsyncTask3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:36.056365] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:36.056381] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.076314] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:36.076327] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337636076310) [2024-09-26 16:00:36.076332] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337635975998, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:36.076351] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223365312314, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:36.076357] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:36.076365] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:36.076369] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:36.076372] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:36.076377] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.076431] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=54][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.076434] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337636076335) [2024-09-26 16:00:36.082668] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=7] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:36.082682] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337635805403]}) [2024-09-26 16:00:36.082727] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D86B-0-0] [lt=6][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337636082669}) [2024-09-26 16:00:36.086576] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:36.086585] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.116847] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:36.116857] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.146160] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=15] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:36.147125] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:36.147132] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.177495] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:36.177501] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.177520] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:36.177527] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337636177518) [2024-09-26 16:00:36.177532] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337636076335, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:36.177548] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223365413938, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:36.177553] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:36.177560] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:36.177564] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:36.177567] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:36.177572] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.177574] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.177576] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337636177535) [2024-09-26 16:00:36.183267] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337635805403]}) [2024-09-26 16:00:36.190200] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:36.190300] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=50, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:36.190932] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=471855, clean_num=31457) [2024-09-26 16:00:36.197939] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:00:36.197954] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=13] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:36.197964] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=5] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:36.207639] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.207645] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.237305] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:36.237327] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=21][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:36.237333] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:36.237349] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:36.237352] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:36.237355] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:36.237358] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:36.237372] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=13][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:36.237374] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:36.237376] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:36.237378] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:36.237380] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:36.237382] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:36.237384] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:36.237391] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:36.237393] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:36.237399] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:36.237402] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:36.237404] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:36.237407] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:36.237410] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:36.237416] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:36.237421] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:36.237423] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:36.237424] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:36.237430] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:36.237433] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:36.237438] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=5][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:36.237441] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:36.237443] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:36.237445] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:36.237448] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337636237235, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:36.237451] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:36.237454] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:36.237483] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:36.237504] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=21][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:36.237508] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=4][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:36.237510] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:36.237513] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:36.237516] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:36.237518] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588C-0-0] [lt=1][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:36.237732] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:36.237739] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.268264] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.268272] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.278024] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:36.278034] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337636278021) [2024-09-26 16:00:36.278039] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337636177535, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:36.278055] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223365512576, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:36.278060] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:36.278068] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:36.278072] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:36.278074] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:36.278080] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.278082] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.278083] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337636278042) [2024-09-26 16:00:36.283568] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337635805403]}) [2024-09-26 16:00:36.298535] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337636226233, now=[mts=1727337635297550]) [2024-09-26 16:00:36.298546] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337636226233, now=[mts=1727337635297550], now0=[mts=1727337635297550], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:36.298553] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92691916e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337635296865, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:36.298569] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=14][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92691916e0}) [2024-09-26 16:00:36.298576] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92691916e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337635296865, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:36.298587] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.298593] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.298584] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92691916e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337635296865, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337634296454, use_das=false, nested_level=0, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92691916e0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:36.298611] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=26][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:36.298616] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:36.298618] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:36.298621] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:36.298623] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:36.298629] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=5][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:36.298633] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:36.298639] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:36.298641] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:36.298642] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:36.298647] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:36.298650] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2002196) [2024-09-26 16:00:36.298653] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:36.298656] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:36.298658] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:36.298660] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:36.298663] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:36.298668] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:36.298696] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:36.298701] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=4][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:36.298704] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:36.298706] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=1][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:36.298709] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:36.298711] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:36.298819] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C78-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:36.298827] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:36.298830] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:36.298844] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=1] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:36.298850] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:36.299124] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:36.299131] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=7][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:36.306456] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.306466] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337636306454], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:36.306472] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337636306454], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:36.306476] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=5509, get_gts_with_stc_cnt=10586, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-09-26 16:00:36.313211] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D390A-0-0] [lt=8][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:36.316106] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1727337636316105, bytes: 14617276, bw: 0.003353 MB/s, add_ts: 1006977, add_bytes: 3540 [2024-09-26 16:00:36.328796] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.328812] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.346393] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:36.359119] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:36.359129] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.378608] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:36.378621] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337636378605) [2024-09-26 16:00:36.378627] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337636278041, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:36.378651] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:36.378686] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:36.378692] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:36.379027] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:36.379039] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:36.379044] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:36.379048] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:36.379052] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:36.379054] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:36.379058] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:36.379060] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:36.379062] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:36.379064] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:36.379066] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:36.379068] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:36.379070] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:36.379073] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:36.379080] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:36.379083] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:36.379085] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:36.379087] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:36.379090] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:36.379092] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:36.379094] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:36.379102] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=6][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:36.379108] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:36.379110] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:36.379112] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:36.379117] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:36.379120] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:36.379122] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885747-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:36.379124] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:36.379127] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:36.379129] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:36.379131] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337636378953, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:36.379136] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:36.379139] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:36.379166] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:36.379172] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337636379171, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=496, wlock_time=20, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:36.379179] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:36.379182] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:36.379197] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223365615198, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:36.379204] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:36.379210] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:36.379214] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:36.379216] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:36.379221] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.379223] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.379225] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337636379185) [2024-09-26 16:00:36.383495] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=7] current monitor number(seq_=-1) [2024-09-26 16:00:36.384512] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:36.384508] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337636306454]}) [2024-09-26 16:00:36.389422] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.389431] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.391441] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:36.391562] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=51, refresh_score_time=27, wash_time=1) [2024-09-26 16:00:36.392163] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=503312, clean_num=31457) [2024-09-26 16:00:36.419650] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29] success to notify_need_writing_throttling True [2024-09-26 16:00:36.419662] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.449920] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.449931] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.451870] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=24] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.451874] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.451879] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.451880] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.451884] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.451885] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.452819] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.452830] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.452834] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.452836] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.452839] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.452842] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.453797] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.453804] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.453806] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.453823] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.453828] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.453832] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.454797] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.454803] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.454804] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.454807] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.454807] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.454809] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.455467] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.455470] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.455473] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.455491] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.455495] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.455497] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.456737] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.456745] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.456745] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.456748] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.456748] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.456750] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.457556] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.457560] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.457562] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.457564] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.457567] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.457568] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.458709] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.458712] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.458713] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.458715] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.458716] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.458717] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:36.459680] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.459685] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.460646] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.460664] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.461628] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.461629] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.462643] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.462648] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.463577] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.463589] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.464565] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.464584] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.465078] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.465166] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.466513] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.466520] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.467481] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.467486] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.468062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.468064] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.469472] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.469481] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.470440] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.470462] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.471437] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.471459] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.472362] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.472369] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.473360] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.473367] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.474349] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.474365] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.474878] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.474882] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.476270] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.476287] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.476902] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.476909] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.478234] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.478243] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.478660] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223365714707, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:36.478670] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:36.478678] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:36.478683] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:36.478686] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:36.478692] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.478694] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.478695] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337636478644) [2024-09-26 16:00:36.478765] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.478772] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.480188] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.480195] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.480250] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.480255] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.481164] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.481177] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.482173] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.482176] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.483145] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.483150] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.484120] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.484137] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.484640] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.484651] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337636306454]}) [2024-09-26 16:00:36.484710] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.486057] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.486056] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.487015] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.487028] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.488015] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.488017] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.488540] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.488593] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.489965] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.489968] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.490745] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.490784] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.491907] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.491909] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.492483] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.492484] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.493897] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.493909] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.494823] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.494849] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.495824] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.495826] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.496786] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.496825] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.497733] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.497750] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.498755] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.498757] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.499720] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.499738] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:36.510486] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:36.510492] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.527313] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185901-0-0] [lt=33][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:36.527325] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185901-0-0] [lt=11][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:36.527334] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185901-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:36.527339] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185901-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:36.540590] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:36.540601] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.540763] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6385, mem_used=12478464) [2024-09-26 16:00:36.546493] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=55] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:36.562223] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=8] gc stale ls task succ [2024-09-26 16:00:36.571073] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:36.571082] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.578805] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:36.578831] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337636578800) [2024-09-26 16:00:36.578839] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337636378649, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:36.578863] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223365814284, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:36.578872] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:36.578883] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:36.578890] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:36.578894] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:36.578903] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.578906] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.578909] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337636578843) [2024-09-26 16:00:36.584688] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337636306454]}) [2024-09-26 16:00:36.593640] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:36.593745] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=55, refresh_score_time=13, wash_time=2) [2024-09-26 16:00:36.594346] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=534769, clean_num=31457) [2024-09-26 16:00:36.601390] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:36.601417] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.601445] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.601436] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.601924] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.601944] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.609144] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855E1-0-0] [lt=2] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:36.631613] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=37] success to notify_need_writing_throttling True [2024-09-26 16:00:36.631622] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.657986] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=4] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:36.658022] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:36.661852] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.661859] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.679464] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:36.679476] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337636679460) [2024-09-26 16:00:36.679482] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337636578843, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:36.679491] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:36.679500] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223365915943, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:36.679506] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:36.679524] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:36.679529] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:36.679532] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:36.679540] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.679543] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.679545] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337636679485) [2024-09-26 16:00:36.685337] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337636306454]}) [2024-09-26 16:00:36.692176] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.692184] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.718567] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4184, dropped:505, tid:2876}]) [2024-09-26 16:00:36.723408] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:36.723449] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.734144] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=20] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:36.746823] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:36.753549] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24] success to notify_need_writing_throttling True [2024-09-26 16:00:36.753558] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.754624] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1727337636754622, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007747, add_bytes: 0 [2024-09-26 16:00:36.780043] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:36.780070] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337636780038) [2024-09-26 16:00:36.780078] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337636679485, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:36.780097] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223366016555, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:36.780104] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:36.780112] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:36.780118] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:36.780120] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:36.780127] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.780129] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.780131] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337636780081) [2024-09-26 16:00:36.783954] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:36.783962] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.786892] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337636306454]}) [2024-09-26 16:00:36.794817] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:36.794923] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash time detail, (compute_wash_size_time=54, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:36.795554] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=566226, clean_num=31457) [2024-09-26 16:00:36.807553] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:36.807563] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337636807551], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:36.807583] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337636807551], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:36.807590] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:36.807594] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:36.808339] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [2915][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=5] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-09-26 16:00:36.814241] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:36.814257] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.820140] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.820150] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.820149] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.820154] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.820191] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.822085] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.822088] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.822099] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.822106] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.822132] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.822146] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:36.844559] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:36.844566] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.865038] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337636793290, now=[mts=1727337635864032]) [2024-09-26 16:00:36.865046] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337636793290, now=[mts=1727337635864032], now0=[mts=1727337635864032], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:36.865053] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a3116d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500714, assoc_session_id:3221500714, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337635863943, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:36.865082] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=28][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22f01f8, id:3221500714, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3116d80}) [2024-09-26 16:00:36.865091] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500714, *tx_desc={this:0x7f92a3116d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500714, assoc_session_id:3221500714, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337635863943, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:36.865101] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=9] start stmt(ret=-4283, auto_commit=true, session_id=3221500714, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3116d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500714, assoc_session_id:3221500714, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337635863943, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337634863495, use_das=false, nested_level=0, session={this:0x7f92a22f01f8, id:3221500714, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3116d80}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:36.865128] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=26][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:36.865133] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:36.865136] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:36.865138] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:36.865141] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:36.865146] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:36.865150] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:36.865157] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:36.865159] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:36.865161] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:36.865166] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=1][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:36.865171] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=4] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2001676) [2024-09-26 16:00:36.865174] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885081-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:36.865177] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:36.865180] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:36.865182] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:36.865185] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8be0, start=1727337634863492, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:36.865190] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:00:36.865194] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:36.865226] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:36.865229] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:36.865232] WDIAG [SHARE] get_snapshot_gc_scn (ob_global_stat_proxy.cpp:164) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:36.865234] WDIAG [STORAGE] get_global_info (ob_tenant_freeze_info_mgr.cpp:811) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to get global info(ret=-4012, tenant_id=1) [2024-09-26 16:00:36.865236] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:954) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] failed to get global info(ret=-4012) [2024-09-26 16:00:36.865251] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1008) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4012] fail to try update info(tmp_ret=-4012, tmp_ret="OB_TIMEOUT") [2024-09-26 16:00:36.874787] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.874802] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.880664] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:36.880674] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337636880661) [2024-09-26 16:00:36.880706] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337636780081, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:36.880712] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:36.880721] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:36.880726] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:36.881044] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:36.881055] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:36.881058] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:36.881062] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:36.881068] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=5][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:36.881070] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:36.881073] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:36.881076] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:36.881078] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:36.881079] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:36.881081] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:36.881083] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:36.881086] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:36.881088] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:36.881096] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:36.881099] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:36.881102] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:36.881104] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:36.881106] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:36.881109] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:36.881111] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:36.881116] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:36.881123] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:36.881125] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:36.881126] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:36.881131] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:36.881134] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:36.881137] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885748-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:36.881139] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:36.881144] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:36.881146] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:36.881149] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337636880965, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:36.881152] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:36.881156] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:36.881193] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:36.881197] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337636881197, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=481, wlock_time=13, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:36.881206] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:36.881209] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:36.881224] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223366117133, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:36.881229] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:36.881237] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:36.881241] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:36.881243] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:36.881247] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.881250] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.881252] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337636881212) [2024-09-26 16:00:36.887482] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337636807551]}) [2024-09-26 16:00:36.893391] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=15] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:36.905076] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.905082] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.935402] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:36.935409] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.935427] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:36.947043] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:36.966589] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:36.966614] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.971522] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] tx gc loop thread is running(MTL_ID()=1) [2024-09-26 16:00:36.971531] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try gc retain ctx [2024-09-26 16:00:36.971935] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-09-26 16:00:36.971964] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:36.982278] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:36.982288] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:36.982295] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337636982275) [2024-09-26 16:00:36.982299] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337636880710, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:36.982314] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223366218781, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:36.982320] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:36.982327] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:36.982332] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:36.982337] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:36.982342] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.982344] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:36.982346] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337636982302) [2024-09-26 16:00:36.988099] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337636807551]}) [2024-09-26 16:00:36.993998] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=6] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService13ObRestartTaskE) [2024-09-26 16:00:36.994025] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] [RS_MGR] new master rootserver found(rootservice="192.168.0.45:2882", cluster_id=1) [2024-09-26 16:00:36.994043] INFO [SHARE.PT] update_leader_replica_ (ob_inmemory_ls_table.cpp:230) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] add leader replica for sys ls(new_replica={modify_time_us:1727337636994042, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"192.168.0.45:2882", sql_port:2881, role:1, member_list:[{server:"192.168.0.45:2882", timestamp:1}], replica_type:0, proposal_id:374, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, ls_info_={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:1727337636994042, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"192.168.0.45:2882", sql_port:2881, role:1, member_list:[{server:"192.168.0.45:2882", timestamp:1}], replica_type:0, proposal_id:374, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-09-26 16:00:36.994066] INFO [SHARE.PT] update (ob_inmemory_ls_table.cpp:145) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] update ls replica(ret=0, ret="OB_SUCCESS", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"192.168.0.45:2882", sql_port:2881, role:1, member_list:[{server:"192.168.0.45:2882", timestamp:1}], replica_type:0, proposal_id:374, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:1727337636994042, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"192.168.0.45:2882", sql_port:2881, role:1, member_list:[{server:"192.168.0.45:2882", timestamp:1}], replica_type:0, proposal_id:374, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-09-26 16:00:36.994086] INFO [SHARE] store (ob_rs_mgr.cpp:128) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] store rs list succeed(agent=0, addr_list=[{server:"192.168.0.45:2882", role:1, sql_port:2881}], force=true) [2024-09-26 16:00:36.994365] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:36.994370] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:36.996025] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:36.996147] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=50, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:36.996730] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:36.996739] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:36.996778] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=597683, clean_num=31457) [2024-09-26 16:00:37.001991] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] new mysql sessid created(conn.sessid_=3221500724, support_ssl=false) [2024-09-26 16:00:37.002027] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221500724, sess.client_addr_="127.0.0.1:36334") [2024-09-26 16:00:37.002034] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f92be35cf90, session_id:3221500724, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:37.002091] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:37.002101] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=5] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:37.002137] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9E-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:37.002297] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9E-0-0] [lt=4][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:37.002320] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9E-0-0] [lt=22][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"A4"K6h.H`*N,.BBdPFL,"}, ret=-4043) [2024-09-26 16:00:37.002331] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9E-0-0] [lt=8][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:37.002334] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9E-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:37.002341] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9E-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:37.002363] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9E-0-0] [lt=5] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500724, proxy_sessid:0}) [2024-09-26 16:00:37.002368] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9E-0-0] [lt=4] mark session id unused(sessid=3221500724) [2024-09-26 16:00:37.002371] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9E-0-0] [lt=2][errcode=0] server close connection(sessid=3221500724, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:37.002376] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9E-0-0] [lt=5][errcode=-4018] get session fail(ret=-4018, sessid=3221500724, proxy_sessid=0) [2024-09-26 16:00:37.002379] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9E-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:37.002381] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9E-0-0] [lt=1] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500724, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:37.002434] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f92be35cf90, session_id:3221500724, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337637001851, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:37.002444] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] kill and revert session(conn.sessid_=3221500724, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:37.002466] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] can close safely, do destroy(*s={this:0x7f92be35cf90, session_id:3221500724, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337637001851, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:37.002471] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500724, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:37.024350] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185902-0-0] [lt=47][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:37.024359] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185902-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:37.024367] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185902-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:37.024372] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185902-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:37.027152] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:37.027159] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.036941] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10707.00) [2024-09-26 16:00:37.036951] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:37.036962] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:37.036965] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:37.039551] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:37.039562] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:37.039565] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:37.039569] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:37.039572] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:37.039574] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:37.039577] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:37.039603] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=17][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:37.039606] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:37.039608] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:37.039610] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:37.039613] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:37.039615] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:37.039618] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:37.039624] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:37.039626] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:37.039629] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:37.039631] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:37.039633] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:37.039635] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:37.039638] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:37.039645] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:37.039650] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=3][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:37.039658] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587E-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:37.050061] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=5] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:37.050402] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E56-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:37.050414] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E56-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:37.050439] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E56-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:37.057283] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:37.057294] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.082808] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=7] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:37.082826] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:37.082841] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337637082823) [2024-09-26 16:00:37.082846] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337636982302, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:37.082867] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223366318338, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:37.082882] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:37.082894] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:37.082899] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:37.082902] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:37.082908] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.082912] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.082915] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337637082851) [2024-09-26 16:00:37.087757] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:37.087767] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.088365] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337636807551]}) [2024-09-26 16:00:37.088411] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D877-0-0] [lt=12][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337637088358}) [2024-09-26 16:00:37.118956] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:37.118965] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.120959] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:51, tid:2464}]) [2024-09-26 16:00:37.147276] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:37.150205] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=60] success to notify_need_writing_throttling True [2024-09-26 16:00:37.150215] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.180494] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:37.180502] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.183467] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:37.183480] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337637183464) [2024-09-26 16:00:37.183495] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337637082851, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:37.183525] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=223366418968, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:37.183537] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:37.183545] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:37.183552] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:37.183555] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:37.183560] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.183562] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.183564] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337637183511) [2024-09-26 16:00:37.189279] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337636807551]}) [2024-09-26 16:00:37.197227] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:37.197330] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=53, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:37.197930] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=629140, clean_num=31457) [2024-09-26 16:00:37.202191] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:37.202204] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:37.202209] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:37.211772] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:37.211781] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.238128] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:37.238142] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:37.238146] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:37.238149] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:37.238153] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:37.238155] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:37.238159] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:37.238161] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:37.238163] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:37.238164] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:37.238166] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:37.238168] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:37.238171] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:37.238173] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:37.238179] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:37.238182] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:37.238184] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:37.238186] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:37.238189] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:37.238191] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:37.238194] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:37.238202] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=6][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:37.238220] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:37.238223] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:37.238224] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:37.238230] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:37.238234] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:37.238236] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:37.238239] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:37.238241] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=1][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:37.238246] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:37.238249] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337637238057, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:37.238252] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:37.238255] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:37.238283] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:37.238288] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=5][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:37.238291] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:37.238293] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:37.238296] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:37.238299] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:37.238301] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588D-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:37.242046] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:37.242055] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.272282] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:37.272290] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.284036] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:37.284046] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337637284032) [2024-09-26 16:00:37.284052] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337637183511, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:37.284078] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223366518591, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:37.284088] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:37.284095] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:37.284100] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:37.284106] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:37.284112] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.284114] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.284116] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337637284054) [2024-09-26 16:00:37.289879] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337636807551]}) [2024-09-26 16:00:37.299651] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=18][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337638228855, now=[mts=1727337636299122]) [2024-09-26 16:00:37.299667] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=16][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337638228855, now=[mts=1727337636299122], now0=[mts=1727337636299122], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:37.299674] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269193480, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337636298521, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:37.299693] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=18][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269193480}) [2024-09-26 16:00:37.299701] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269193480, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337636298521, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:37.299708] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269193480, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337636298521, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636299096, use_das=false, nested_level=0, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269193480}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:37.299725] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:37.299730] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:37.299732] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:37.299735] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:37.299737] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:37.299740] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:37.299745] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=999111, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337638298855) [2024-09-26 16:00:37.299748] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:37.299754] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:37.299756] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:37.299757] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:37.299762] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:37.299824] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:37.299830] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:37.302596] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:37.302602] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.308571] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.308580] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337637308569], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:37.308587] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337637308569], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:37.319224] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D3981-0-0] [lt=23][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:37.324075] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337637324074, bytes: 14621098, bw: 0.003616 MB/s, add_ts: 1007969, add_bytes: 3822 [2024-09-26 16:00:37.332871] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:37.332878] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.347519] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:37.363119] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:37.363129] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.383621] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2024-09-26 16:00:37.384110] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:37.384120] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337637384107) [2024-09-26 16:00:37.384128] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337637284054, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:37.384134] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:37.384142] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:37.384145] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:37.384492] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:37.384504] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:37.384508] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:37.384527] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:37.384534] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=6][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:37.384537] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:37.384541] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:37.384543] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:37.384545] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:37.384547] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:37.384556] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:37.384558] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:37.384561] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:37.384557] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=12] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:37.384563] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:37.384569] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:37.384572] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:37.384575] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:37.384577] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:37.384582] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:37.384585] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:37.384588] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:37.384593] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:37.384599] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:37.384601] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:37.384602] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:37.384607] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:37.384613] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=5][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:37.384615] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885749-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:37.384618] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:37.384620] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:37.384622] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:37.384625] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337637384417, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:37.384628] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:37.384631] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:37.384662] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:37.384667] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337637384666, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=527, wlock_time=8, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:37.384675] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:37.384679] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:37.384694] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223366620438, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:37.384698] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:37.384704] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:37.384710] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:37.384723] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:37.384728] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.384730] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.384732] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337637384682) [2024-09-26 16:00:37.387782] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:37.387790] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=6] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:37.387795] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=4][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:37.387799] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:37.387801] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:37.387804] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:37.387806] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:37.387808] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:37.387811] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:37.387813] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:37.387814] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:37.387816] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:37.387818] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:37.387819] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:37.387821] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:37.387823] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:37.387826] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:37.387828] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:37.387832] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:37.387834] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:37.387836] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-09-26 16:00:37.387841] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=2][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:37.387844] WDIAG [SERVER] do_process (obmp_query.cpp:771) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=3][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-09-26 16:00:37.387849] WDIAG [SERVER] do_process (obmp_query.cpp:903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, session={this:0x7f92a230bdf8, id:3221487638, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-09-26 16:00:37.387859] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4D-0-0] [lt=7] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:37.390041] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337637308569]}) [2024-09-26 16:00:37.393500] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:37.393519] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.398432] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:37.398531] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:37.399150] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=660597, clean_num=31457) [2024-09-26 16:00:37.423732] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:37.423750] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.453960] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:37.453971] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.453990] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=15] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.453997] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.454002] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.454017] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.454024] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.454029] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.454612] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.454617] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.454619] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.454621] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.454623] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.454623] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.455916] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.455920] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.455921] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.455923] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.455924] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.455925] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.456502] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.456504] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.456507] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.456507] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.456509] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.456509] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.457865] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.457868] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.457870] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.457866] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.457873] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.457877] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.458397] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.458401] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.458402] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.458403] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.458404] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.458406] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.459829] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.459835] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.459831] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.459839] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.459839] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.459845] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.460380] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.460382] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.460385] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.460387] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.460388] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.460389] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:37.461757] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.461773] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.462384] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.462386] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.463721] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.463727] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.464746] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.464756] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.465692] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.465693] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.466639] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.466659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.467647] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.467663] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.468606] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.468608] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.469591] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.469593] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.470170] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.470170] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.471540] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.471556] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.472547] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] ====== tenant freeze timer task ====== [2024-09-26 16:00:37.472555] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:37.472560] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.472585] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337635622676, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337635622633, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337635622724, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:37.472612] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:37.472635] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.472675] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D7-0-0] [lt=3] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:37.472683] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D7-0-0] [lt=5] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:37.472688] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D7-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:37.472691] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D7-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:37.472694] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D7-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:37.472697] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D7-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:37.472699] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D7-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:37.472708] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D7-0-0] [lt=9] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:37.472711] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D7-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:37.472713] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D7-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:37.472716] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D7-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:37.472744] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851D7-0-0] [lt=4] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:37.473131] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.473206] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.474483] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.474641] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.475457] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.475461] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.476418] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.476457] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.476952] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.477018] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.478381] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.478393] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.479358] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.479383] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.480320] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.480334] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.481298] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.481318] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.481991] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.482017] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.483264] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.483271] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.484094] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.484107] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:37.484111] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.484127] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.484200] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:37.484210] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337637484197) [2024-09-26 16:00:37.484215] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337637384131, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:37.484231] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223366719741, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:37.484236] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:37.484243] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:37.484248] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:37.484250] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:37.484255] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.484257] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.484259] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337637484217) [2024-09-26 16:00:37.484620] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.484705] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.485197] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.485283] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.486191] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.486195] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.486568] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=5][errcode=-4018] try free send task failed(ret=-4018) [2024-09-26 16:00:37.486719] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.486725] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.488166] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.488169] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.489135] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.489146] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.490093] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.490102] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.490104] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337637308569]}) [2024-09-26 16:00:37.490678] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.490702] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.491301] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.491304] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.493036] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.493054] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.493965] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.494008] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.495014] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.495055] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.495581] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.495586] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.496939] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.496939] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.497892] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.497910] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.499864] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.499921] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.500832] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.500841] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.501821] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.501821] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:37.515516] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:37.515524] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.526413] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185903-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:37.526422] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185903-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:37.526430] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185903-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:37.526436] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185903-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:37.540902] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=41] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6393, mem_used=12478464) [2024-09-26 16:00:37.545596] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27] success to notify_need_writing_throttling True [2024-09-26 16:00:37.545604] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.547711] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:37.562235] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:00:37.575770] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:37.575778] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.584833] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:37.584866] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337637584829) [2024-09-26 16:00:37.584873] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337637484217, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:37.584893] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223366820398, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:37.584909] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:37.584919] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:37.584930] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:37.584934] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:37.584943] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.584948] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.584952] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337637584877) [2024-09-26 16:00:37.590702] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337637308569]}) [2024-09-26 16:00:37.599636] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:37.599736] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=56, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:37.600326] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=692054, clean_num=31457) [2024-09-26 16:00:37.602464] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.602475] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.603769] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.603783] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.607536] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:45) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2] dump dtl interm result cost(us)(dump_cost=13902, ret=0, interm count=0, dump count=343) [2024-09-26 16:00:37.612317] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:37.612323] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.621222] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:60) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10] clear dtl interm result cost(us)(clear_cost=13675, ret=0, expire_keys_.count()=0, dump count=343, interm count=0, clean count=0) [2024-09-26 16:00:37.621664] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=14] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:37.621674] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:37.621678] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:37.621685] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=6][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:37.621689] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:37.621691] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:37.621695] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:37.621698] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:37.621699] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:37.621701] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:37.621704] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:37.621706] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:37.621710] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:37.621713] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:37.621720] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:37.621722] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:37.621724] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:37.621727] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:37.621729] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:37.621732] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:37.621736] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:37.621743] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:37.621749] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:37.621751] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:37.621752] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:37.621758] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:37.621764] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD2-0-0] [lt=5][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:37.621767] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2458][ServerGTimer][T0][YB42C0A8002D-0006230056A84DD2-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:37.621770] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:37.621772] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:37.621774] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:37.621776] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337637621591, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:37.621792] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:37.621828] WDIAG [SHARE] refresh (ob_all_server_tracer.cpp:568) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] fail to get servers_info(ret=-5019, ret="OB_TABLE_NOT_EXIST", GCTX.sql_proxy_=0x55c7486d2e88) [2024-09-26 16:00:37.621832] WDIAG [SHARE] runTimerTask (ob_all_server_tracer.cpp:626) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] fail to refresh all server map(ret=-5019) [2024-09-26 16:00:37.643452] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=38] success to notify_need_writing_throttling True [2024-09-26 16:00:37.643472] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.658096] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:37.662967] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:37.673737] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:37.673753] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.685234] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:37.685247] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337637685231) [2024-09-26 16:00:37.685253] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337637584877, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:37.685268] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223366920926, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:37.685265] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:37.685273] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:37.685281] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:37.685286] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:37.685288] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:37.685293] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.685295] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.685297] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337637685255) [2024-09-26 16:00:37.691354] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337637308569]}) [2024-09-26 16:00:37.704401] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:37.704411] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.735262] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:37.735270] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.748238] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=707760, replace_num=15728) [2024-09-26 16:00:37.748264] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:37.762622] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1727337637762621, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007999, add_bytes: 0 [2024-09-26 16:00:37.765573] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:37.765582] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.786095] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:37.786110] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337637786092) [2024-09-26 16:00:37.786116] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337637685255, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:37.786134] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223367021554, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:37.786143] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:37.786151] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:37.786156] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:37.786159] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:37.786164] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.786169] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.786171] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337637786119) [2024-09-26 16:00:37.791908] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337637308569]}) [2024-09-26 16:00:37.796584] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:37.796592] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.800565] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:37.800683] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=72, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:37.801291] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=723511, clean_num=31457) [2024-09-26 16:00:37.808977] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:37.808990] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337637808974], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:37.808998] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337637808974], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:37.809002] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:37.809006] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:37.821208] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.821243] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.821259] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.821260] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.821269] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=11] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.821269] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.821276] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.821281] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.821637] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.821648] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.821654] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:37.827081] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=34] success to notify_need_writing_throttling True [2024-09-26 16:00:37.827090] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.858390] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:37.858398] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.886669] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:37.886692] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337637886649) [2024-09-26 16:00:37.886698] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337637786119, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:37.886704] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:37.886714] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:37.886717] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:37.887066] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:37.887076] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:37.887080] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:37.887084] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:37.887088] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:37.887114] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=25][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:37.887119] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:37.887121] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:37.887123] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:37.887125] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:37.887127] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:37.887130] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:37.887132] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:37.887135] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:37.887142] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:37.887147] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:37.887150] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:37.887153] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:37.887155] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:37.887157] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:37.887160] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:37.887165] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:37.887170] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:37.887172] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:37.887173] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:37.887179] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:37.887182] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:37.887184] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574A-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:37.887186] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:37.887189] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:37.887191] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:37.887193] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337637886977, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:37.887197] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:37.887199] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:37.887229] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:37.887232] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337637887231, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=521, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:37.887240] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:37.887259] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:37.887275] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223367122141, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:37.887280] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:37.887286] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:37.887290] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:37.887292] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:37.887297] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.887299] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.887301] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337637887263) [2024-09-26 16:00:37.889006] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:37.889012] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.892534] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337637808974]}) [2024-09-26 16:00:37.919870] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:37.919876] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.948293] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=4] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:37.950117] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:37.950123] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.950132] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:37.972646] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:37.980463] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:37.980471] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:37.987256] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:37.987293] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:37.987301] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337637987253) [2024-09-26 16:00:37.987306] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337637886701, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:37.987323] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223367221536, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:37.987329] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:37.987337] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:37.987343] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:37.987346] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:37.987354] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.987356] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:37.987358] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337637987309) [2024-09-26 16:00:37.993169] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337637808974]}) [2024-09-26 16:00:37.995098] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337636994363]) [2024-09-26 16:00:37.995108] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337636994363], now0=[mts=1727337636994363], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:37.995125] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337636993948, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:37.995151] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=25][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:37.995159] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337636993948, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:37.995167] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337636993948, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:37.995184] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:37.995189] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:37.995192] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:37.995195] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:37.995197] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:37.995201] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:37.995206] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=28999126, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337666994332) [2024-09-26 16:00:37.995210] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:37.995217] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:37.995219] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:37.995220] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:37.995226] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:37.995262] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:37.995265] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:38.002020] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:38.002131] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=15, wash_time=10) [2024-09-26 16:00:38.002744] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=754968, clean_num=31457) [2024-09-26 16:00:38.002961] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221500728, support_ssl=false) [2024-09-26 16:00:38.002993] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221500728, sess.client_addr_="127.0.0.1:36336") [2024-09-26 16:00:38.003000] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f9264004030, session_id:3221500728, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:38.003021] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:38.003029] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:38.003075] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9E-0-0] [lt=28] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:38.003323] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9E-0-0] [lt=7][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:38.003331] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9E-0-0] [lt=7][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"!n*"js"K4cuF:.x6C+N("}, ret=-4043) [2024-09-26 16:00:38.003337] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9E-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:38.003341] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9E-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:38.003348] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9E-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:38.003371] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9E-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500728, proxy_sessid:0}) [2024-09-26 16:00:38.003375] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9E-0-0] [lt=4] mark session id unused(sessid=3221500728) [2024-09-26 16:00:38.003378] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9E-0-0] [lt=2][errcode=0] server close connection(sessid=3221500728, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:38.003382] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9E-0-0] [lt=4][errcode=-4018] get session fail(ret=-4018, sessid=3221500728, proxy_sessid=0) [2024-09-26 16:00:38.003384] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9E-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:38.003386] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9E-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500728, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:38.003447] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4015] close sql sock by user req(*s={this:0x7f9264004030, session_id:3221500728, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337638001880, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:38.003485] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=37] kill and revert session(conn.sessid_=3221500728, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:38.003489] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=4] can close safely, do destroy(*s={this:0x7f9264004030, session_id:3221500728, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337638001880, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:38.003493] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500728, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:38.011664] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:38.011671] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.022785] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:38.022798] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-09-26 16:00:38.022803] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:38.022806] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-09-26 16:00:38.022810] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:38.022812] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:38.022815] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-09-26 16:00:38.022817] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:38.022819] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:38.022822] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:38.022824] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=1][errcode=-5019] resolve table failed(ret=-5019) [2024-09-26 16:00:38.022827] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:2822) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=3][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-09-26 16:00:38.022830] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2676) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=2][errcode=-5019] resolve joined table failed(ret=-5019) [2024-09-26 16:00:38.022831] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:38.022833] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:38.022836] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:38.022838] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:38.022847] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:38.022849] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:38.022853] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:38.022854] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:38.022857] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-09-26 16:00:38.022860] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:38.022862] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:38.022868] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:38.022873] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:38.022875] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:38.022876] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:38.022882] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:38.022888] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9E-0-0] [lt=6][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:38.022895] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:398) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=0] start to print blacklist info [2024-09-26 16:00:38.022916] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3] ls blacklist refresh finish(cost_time=454) [2024-09-26 16:00:38.028183] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185904-0-0] [lt=37][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:38.028194] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185904-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:38.028209] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185904-0-0] [lt=14][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:38.028234] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185904-0-0] [lt=24][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:38.040277] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587F-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:38.040289] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587F-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:38.040310] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8587F-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:38.042053] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:38.042059] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.043897] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 12141.00) [2024-09-26 16:00:38.043905] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:38.043917] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:38.043920] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:38.050560] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=10] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:38.050836] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E58-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:38.050846] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E58-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:38.050866] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E58-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:38.072225] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:38.072236] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.082973] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=10] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:38.087847] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:38.087868] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337638087843) [2024-09-26 16:00:38.087874] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337637987309, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:38.087894] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223367323382, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:38.087903] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:38.087911] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:38.087928] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:38.087931] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:38.087938] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.087940] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.087942] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337638087877) [2024-09-26 16:00:38.093713] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337637808974]}) [2024-09-26 16:00:38.093725] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D883-0-0] [lt=9][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337638093709}) [2024-09-26 16:00:38.102972] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:38.102981] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.133409] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:38.133416] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.148462] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:38.163567] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:38.163575] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.188514] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:38.188525] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337638188508) [2024-09-26 16:00:38.188531] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337638087877, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:38.188557] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223367424018, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:38.188567] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:38.188575] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:38.188591] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:38.188595] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:38.188601] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.188603] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.188605] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337638188534) [2024-09-26 16:00:38.194323] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:38.194330] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.194397] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337637808974]}) [2024-09-26 16:00:38.198211] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:00:38.198217] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=5] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:38.198225] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:38.203247] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:38.203362] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:38.203984] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=786425, clean_num=31457) [2024-09-26 16:00:38.224383] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:38.224390] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.238711] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:38.238722] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:38.238727] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:38.238730] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:38.238733] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:38.238736] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:38.238739] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:38.238741] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:38.238744] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:38.238745] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:38.238747] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:38.238749] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:38.238769] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588E-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:38.254866] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:38.254875] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.285142] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:38.285152] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.289064] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:38.289076] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337638289059) [2024-09-26 16:00:38.289082] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337638188534, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:38.289100] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223367524536, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:38.289119] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:38.289127] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:38.289140] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:38.289145] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:38.289152] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.289154] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.289155] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337638289085) [2024-09-26 16:00:38.294903] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337637808974]}) [2024-09-26 16:00:38.300802] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337638228855, now=[mts=1727337637299823]) [2024-09-26 16:00:38.300812] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=9][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337638228855, now=[mts=1727337637299823], now0=[mts=1727337637299823], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:38.300818] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269193480, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337637298742, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:38.300835] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269193480}) [2024-09-26 16:00:38.300843] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269193480, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337637298742, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:38.300850] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269193480, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337637298742, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636299096, use_das=false, nested_level=0, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269193480}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:38.300867] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:38.300871] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:38.300874] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:38.300877] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:38.300879] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:38.300883] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:38.300887] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:38.300893] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:38.300895] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:38.300897] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:38.300902] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C79-0-0] [lt=1][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:38.300905] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2001808) [2024-09-26 16:00:38.300907] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:38.300910] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:38.300913] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:38.300915] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:38.300917] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:38.300922] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:38.300950] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:38.300952] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:38.300955] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:38.300957] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:38.300961] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:38.300963] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:38.301070] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C79-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:38.301075] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:38.301078] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:38.301118] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=1] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:38.301123] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=4][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:38.301452] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=11][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:38.301457] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:38.309700] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:38.309710] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337638309698], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:38.309718] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337638309698], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:38.309721] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337638309698]}) [2024-09-26 16:00:38.315454] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:38.315457] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD3-0-0] [lt=2][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-09-26 16:00:38.315462] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD3-0-0] [lt=5][errcode=-4002] refresh sever list failed(ret=-4002) [2024-09-26 16:00:38.315464] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD3-0-0] [lt=2][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-09-26 16:00:38.315462] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.320338] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D39E1-0-0] [lt=19][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:38.326216] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:109, tid:2919}]) [2024-09-26 16:00:38.332020] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1727337638332019, bytes: 14624638, bw: 0.003349 MB/s, add_ts: 1007945, add_bytes: 3540 [2024-09-26 16:00:38.348544] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:38.348595] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:38.348601] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.366216] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2648) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12] [PALF_DUMP](palf_id=1, self="192.168.0.45:2882", [SlidingWindow]={palf_id:1, self:"192.168.0.45:2882", lsn_allocator:{max_log_id:183389608, max_lsn:629872191939, max_scn:{val:1727114270763524000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:629872191817}, reuse_lsn:{lsn:629872191939}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:629872191939}}, last_submit_lsn:{lsn:629872191817}, last_submit_end_lsn:{lsn:629872191939}, last_submit_log_id:183389608, last_submit_log_pid:19, max_flushed_lsn:{lsn:629872191817}, max_flushed_end_lsn:{lsn:629872191939}, max_flushed_log_pid:19, committed_end_lsn:{lsn:629872191939}, last_slide_log_id:183389608, last_slide_scn:{val:1727114270763524000, v:0}, last_slide_lsn:{lsn:629872191817}, last_slide_end_lsn:{lsn:629872191939}, last_slide_log_pid:19, last_slide_log_accum_checksum:2487403767, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f92b3ff0530}, [StateMgr]={this:0x7f92b3ff42b0, self:"192.168.0.45:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:374}, leader:"192.168.0.45:2882", leader_epoch:195, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:629872191939}, scan_disk_log_finished:true, last_check_start_id:183389609, is_changing_config_with_arb:false, reconfirm_start_time_us:1727336098224413, palf_role_change_cb:0x7f92b3ffab70, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.0.45:2882", alive_paxos_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:374, prev:{version:1, config:{config_version:{proposal_id:373, config_seq:374}, log_sync_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:374, config_seq:375}, log_sync_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:19, prev_lsn:{lsn:629872191817}, prev_mode_pid:374}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:19, prev_lsn:{lsn:629872191817}, prev_end_lsn:{lsn:629872191939}, prev_mode_pid:374}, reconfig_barrier:{prev_log_proposal_id:19, prev_lsn:{lsn:629872191817}, prev_end_lsn:{lsn:629872191939}, prev_mode_pid:374}, persistent_config_version:{proposal_id:374, config_seq:375}, ms_ack_list:0, resend_config_version:{proposal_id:374, config_seq:375}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", register_time_us:1727336095809026, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1727336095809026, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f92b3ff1c70}, [ModeMgr]={palf_id:1, self:"192.168.0.45:2882", applied_mode_meta:{version:1, proposal_id:374, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:374, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:374, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1726163155508348002, v:0}, min_block_id:9357, min_block_min_scn:{val:18446744073709551615, v:3}, base_lsn_for_block_gc:{lsn:627966418944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:374}, log_config_meta:{version:3, proposal_id:374, prev:{version:1, config:{config_version:{proposal_id:373, config_seq:374}, log_sync_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:374, config_seq:375}, log_sync_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:19, prev_lsn:{lsn:629872191817}, prev_mode_pid:374}, log_snapshot_meta:{version:1, base_lsn:{lsn:627966418944}, prev_log_info:{log_id:-1, lsn:{lsn:18446744073709551615}, scn:{val:18446744073709551615, v:3}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:374, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:11440128}, readable_log_tail:{lsn:11440128}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:629912457216}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/observer/store/clog/tenant_1/1/meta", dir_fd:125, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:55664640, block_header_serialize_buf_:0x7f92b3ff6438, flashback_version:0}, log_storage:{log_tail:{lsn:629872191939}, readable_log_tail:{lsn:629872191939}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:9386, palf_id:1}, block_mgr:{log_dir:"/root/observer/store/clog/tenant_1/1/log", dir_fd:128, min_block_id:9357, max_block_id:9387, curr_writable_block_id:9386}, logical_block_size_:67104768, curr_block_writable_size_:40265277, block_header_serialize_buf_:0x7f92b3ff80b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1727336098222784, this:0x7f92b3ff49f0}, [Reconfirm]={palf_id:1, self:"192.168.0.45:2882", state:"FINISHED", new_proposal_id:374, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"192.168.0.45:2882", majority_max_accept_pid:373, majority_max_lsn:{lsn:629872191939}, saved_end_lsn:{lsn:629872191939}, last_submit_prepare_req_time_us:1727336098224520, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f92b3ff4430}) [2024-09-26 16:00:38.366316] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2652) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=97] [PALF_DUMP](palf_id=1, self="192.168.0.45:2882", ack_info_list=[{member:{server:"192.168.0.45:2882", timestamp:1, flag:0}, last_ack_time_us:1727336098526269, last_flushed_end_lsn:{lsn:629872191939}}]) [2024-09-26 16:00:38.366322] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4290) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=4] current log_task status(palf_id=1, self="192.168.0.45:2882", log_id=183389609, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.0.45:2882", timestamp:1, flag:0}, last_ack_time_us:1727336098526269, last_flushed_end_lsn:{lsn:629872191939}}]) [2024-09-26 16:00:38.366333] WDIAG [STORAGE] get_server_region (ob_locality_manager.cpp:399) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4018] fail to get server region(ret=-4018, server="192.168.0.45:2882") [2024-09-26 16:00:38.366338] WDIAG [CLOG] get_server_region (ob_locality_adapter.cpp:61) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4018] get_server_region failed(server="192.168.0.45:2882") [2024-09-26 16:00:38.366342] WDIAG [PALF] update_self_region_ (palf_handle_impl.cpp:408) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4018] get_server_region failed(ret=-4018, this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}) [2024-09-26 16:00:38.366345] WDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2658) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4018] update_region failed(ret=-4018, this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}) [2024-09-26 16:00:38.378914] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:38.378926] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.384763] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:38.386728] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [2645][Blacklist][T0][Y0-0000000000000000-0-0] [lt=7] blacklist_loop exec finished(cost_time=10, is_enabled=true, send_cnt=0) [2024-09-26 16:00:38.389675] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:38.389688] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337638389671) [2024-09-26 16:00:38.389693] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337638289085, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:38.389699] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:38.389709] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:38.389713] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:38.390037] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:38.390048] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:38.390052] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:38.390056] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:38.390060] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:38.390062] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:38.390065] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:38.390068] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:38.390071] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:38.390072] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:38.390075] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:38.390078] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:38.390081] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:38.390083] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:38.390090] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:38.390093] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:38.390095] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:38.390098] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:38.390100] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:38.390102] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:38.390104] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:38.390126] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=8][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:38.390134] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:38.390136] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:38.390138] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:38.390144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:38.390147] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:38.390150] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574B-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:38.390152] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:38.390155] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:38.390157] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:38.390159] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337638389957, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:38.390163] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:38.390165] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:38.390197] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:38.390200] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337638390200, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=495, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:38.390207] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:38.390224] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:38.390244] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223367626355, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:38.390249] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:38.390255] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:38.390260] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:38.390262] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:38.390267] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.390269] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.390271] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337638390228) [2024-09-26 16:00:38.395508] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337638309698]}) [2024-09-26 16:00:38.404465] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:38.404589] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=61, refresh_score_time=26, wash_time=1) [2024-09-26 16:00:38.405218] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=817882, clean_num=31457) [2024-09-26 16:00:38.409198] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:38.409216] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.439469] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:38.439476] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.452162] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:33) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=3] === Run print tenant memory usage task === [2024-09-26 16:00:38.452220] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 413,495,296 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 180,654,080 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 88,080,384 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 12,582,912 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 17,969,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 11,292,672 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 92,430,336 limit= 9,223,372,036,854,775,807 [2024-09-26 16:00:38.452321] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:176) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 180,654,080 used= 173,144,032 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] hold= 33,574,912 used= 33,554,464 count= 1 avg_used= 33,554,464 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 18,894,848 used= 18,874,392 count= 1 avg_used= 18,874,392 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 12,460,032 used= 11,125,608 count= 169 avg_used= 65,832 block_cnt= 169 chunk_cnt= 29 mod=[T]ObSessionDIB [MEMORY] hold= 10,776,576 used= 10,735,904 count= 11 avg_used= 975,991 block_cnt= 11 chunk_cnt= 7 mod=IoControl [MEMORY] hold= 8,585,216 used= 8,566,048 count= 1 avg_used= 8,566,048 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 5,251,072 used= 5,238,784 count= 32 avg_used= 163,712 block_cnt= 32 chunk_cnt= 9 mod=Iteratorget_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6402, mem_used=12478464) [2024-09-26 16:00:38.548875] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:38.560624] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:38.560632] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.562557] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:00:38.589946] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:38.589965] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337638589934) [2024-09-26 16:00:38.589971] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337638389696, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:38.589989] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223367826424, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:38.589995] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:38.590003] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:38.590008] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:38.590012] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:38.590018] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.590022] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.590024] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337638589974) [2024-09-26 16:00:38.591829] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:38.591836] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.596725] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337638309698]}) [2024-09-26 16:00:38.601658] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.601672] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.601764] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:783) [2618][AutoLSLocRpc][T0][YB42C0A8002D-0006230057E85880-0-0] [lt=20] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.0.45:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, location:{server:"192.168.0.45:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:374}}]) [2024-09-26 16:00:38.602569] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.602580] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.606659] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:38.606781] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash time detail, (compute_wash_size_time=62, refresh_score_time=25, wash_time=1) [2024-09-26 16:00:38.607395] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=849339, clean_num=31457) [2024-09-26 16:00:38.622120] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:38.622130] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.652458] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:38.652466] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.658313] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:38.668083] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:38.682683] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:38.682689] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.691492] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:38.691504] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337638691489) [2024-09-26 16:00:38.691510] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337638589974, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:38.691526] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223367927964, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:38.691532] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:38.691539] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:38.691543] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:38.691545] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:38.691551] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.691552] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.691554] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337638691513) [2024-09-26 16:00:38.691846] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:38.691855] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:38.691859] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:38.691862] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:38.691866] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:38.691870] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:38.691874] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:38.691876] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:38.691878] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:38.691879] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:38.691882] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:38.691884] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:38.691886] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:38.691889] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:38.691894] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:38.691897] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:38.691899] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:38.691903] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:38.691906] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, ret=-5019) [2024-09-26 16:00:38.691908] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:38.691910] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:38.691916] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:38.691921] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:38.691923] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:38.691925] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:38.691930] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:38.691933] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:38.691936] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD3-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-09-26 16:00:38.691938] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:38.691940] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:38.691943] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:38.691945] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337638691764, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:38.691948] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:38.691951] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f92b4056738, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, columns_str="zone") [2024-09-26 16:00:38.691957] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:38.691985] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:634) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f92b4056738, where_condition="where svr_ip='192.168.0.45' and svr_port=2882", zone_name_holder=) [2024-09-26 16:00:38.691988] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:567) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:38.691991] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:576) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:38.691993] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:38.691999] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:38.697328] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=39] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337638309698]}) [2024-09-26 16:00:38.713041] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:38.713049] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.744219] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:38.744226] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.749084] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:38.760809] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1073) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=5] replay status is not follower(min_unreplayed_lsn={lsn:629872191939}, base_lsn={lsn:627966418944}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:00:38.760834] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=24] get_replay_process success(id={id:1}, replayed_log_size=1905772995, unreplayed_log_size=0) [2024-09-26 16:00:38.760847] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:159) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=3] dump tenant replay process(tenant_id=1, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=1817, last_replayed_log_size(MB)=1817, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-09-26 16:00:38.769633] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337638769633, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007012, add_bytes: 0 [2024-09-26 16:00:38.774493] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:38.774499] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.787168] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [2910][T1_RFLWorker][T1][YB42C0A8002D-0006230055684B24-0-0] [lt=7] ObRemoteFetchWorker is running(thread_index=0) [2024-09-26 16:00:38.792062] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:38.792073] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337638792059) [2024-09-26 16:00:38.792078] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337638691513, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:38.792103] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223368027599, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:38.792110] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:38.792118] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:38.792132] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:38.792135] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:38.792151] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.792155] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.792157] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337638792081) [2024-09-26 16:00:38.798944] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337638309698]}) [2024-09-26 16:00:38.805736] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:38.805748] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.807821] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:38.807953] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash time detail, (compute_wash_size_time=65, refresh_score_time=23, wash_time=1) [2024-09-26 16:00:38.808565] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=880796, clean_num=31457) [2024-09-26 16:00:38.810735] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:38.810742] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337638810733], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:38.810747] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337638810733], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:38.810762] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:38.810766] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:38.820552] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.820564] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.820569] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.820810] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.820824] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.820830] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.821906] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.821923] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.821988] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.822026] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.822035] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:38.836007] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:38.836022] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.867290] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14] success to notify_need_writing_throttling True [2024-09-26 16:00:38.867299] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.892669] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:38.892682] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337638892666) [2024-09-26 16:00:38.892689] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337638792080, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:38.892695] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:38.892721] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:38.892724] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:38.893073] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:38.893083] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:38.893086] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:38.893090] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:38.893094] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:38.893097] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:38.893100] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:38.893103] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:38.893105] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:38.893108] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:38.893110] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:38.893113] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:38.893116] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:38.893118] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:38.893124] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:38.893127] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:38.893130] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:38.893149] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=18][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:38.893152] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:38.893156] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:38.893159] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:38.893166] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:38.893171] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:38.893174] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:38.893175] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:38.893184] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:38.893187] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:38.893190] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574C-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:38.893193] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:38.893198] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:38.893200] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:38.893203] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337638892996, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:38.893206] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:38.893209] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:38.893239] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:38.893243] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337638893242, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=526, wlock_time=11, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:38.893250] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:38.893253] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:38.893268] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223368129154, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:38.893273] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:38.893279] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:38.893284] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:38.893287] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:38.893292] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.893294] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.893296] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337638893256) [2024-09-26 16:00:38.897587] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:38.897594] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.900485] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337638810733]}) [2024-09-26 16:00:38.920970] INFO [CLOG] run1 (ob_garbage_collector.cpp:1338) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] Garbage Collector is running(seq_=344, gc_interval=10000000) [2024-09-26 16:00:38.920988] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1473) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] self is leader, skip it(ls->get_ls_id()={id:1}) [2024-09-26 16:00:38.920996] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1431) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=4] gc_check_member_list_ cost time(ret=0, time_us=17) [2024-09-26 16:00:38.921000] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=2] execute_gc cost time(ret=0, time_us=0) [2024-09-26 16:00:38.921002] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=1] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-09-26 16:00:38.921006] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2024-09-26 16:00:38.921009] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=2] execute_gc cost time(ret=0, time_us=3) [2024-09-26 16:00:38.921011] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=1] ObSafeDestroyHandler start process [2024-09-26 16:00:38.921013] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=2] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2024-09-26 16:00:38.921021] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2024-09-26 16:00:38.927825] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:38.927832] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.946406] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=20] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:38.949345] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:38.957522] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=5] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-09-26 16:00:38.957533] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-09-26 16:00:38.957537] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-09-26 16:00:38.957540] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-09-26 16:00:38.957543] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:38.957546] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:38.957549] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-09-26 16:00:38.957552] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:38.957554] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:38.957556] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:38.957558] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:38.957560] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:38.957562] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:38.957564] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:38.957570] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:38.957572] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:38.957574] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:38.957576] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:38.957578] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:38.957581] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:38.957583] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:38.957589] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:38.957594] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:38.957596] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:38.957597] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:38.957602] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:38.957605] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:38.957608] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2024-09-26 16:00:38.957610] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:38.957613] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:38.957614] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:38.957616] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337638957440, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:38.957620] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:38.957622] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1111) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:38.957645] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:801) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:38.957648] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:67) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:38.957650] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:436) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:38.957652] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:95) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:38.957654] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:109) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:38.957868] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-09-26 16:00:38.957872] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=4][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2024-09-26 16:00:38.957874] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2024-09-26 16:00:38.957876] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2024-09-26 16:00:38.957878] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:38.957880] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:38.957882] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_tenant' doesn't exist [2024-09-26 16:00:38.957883] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:38.957885] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:38.957886] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:38.957887] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:38.957888] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:38.957890] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:38.957891] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:38.957894] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:38.957895] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:38.957896] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:38.957898] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:38.957899] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:38.957901] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:38.957902] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:38.957905] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:38.957908] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:38.957912] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:38.957913] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:38.957916] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:38.957919] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD4-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:38.957924] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=4][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, aret=-5019, ret=-5019) [2024-09-26 16:00:38.957941] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=17][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:00:38.957944] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:38.957946] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:38.957948] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337638957844, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:00:38.957950] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:38.957952] WDIAG [SHARE] read_tenants (ob_unit_table_operator.cpp:1174) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] execute sql failed(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:38.957966] WDIAG [SHARE] get_tenants (ob_unit_table_operator.cpp:108) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2][errcode=-5019] fail read tenants(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:38.957969] WDIAG [SHARE] get_tenants (ob_unit_getter.cpp:198) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=3][errcode=-5019] ut_operator get_resource_pools failed(ret=-5019) [2024-09-26 16:00:38.957971] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:125) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=1][errcode=-5019] get cluster tenants fail(ret=-5019) [2024-09-26 16:00:38.957982] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:325) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=1] cal_all_part_disk_default_percentage succ(data_dir="/root/observer/store/sstable", clog_dir="/root/observer/store/clog", shared_mode=true, data_disk_total_size=99765694464, data_disk_default_percentage=60, clog_disk_total_size=99765694464, clog_disk_default_percentage=30) [2024-09-26 16:00:38.957986] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:361) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=4] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=99765694464, disk_size=15032385536) [2024-09-26 16:00:38.957989] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=99765694464) [2024-09-26 16:00:38.957992] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=2] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=99765694464, this={dir::"/root/observer/store/clog/log_pool", dir_fd:15, meta_fd:16, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:9388, max_block_id:9581, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2024-09-26 16:00:38.957985] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:38.957997] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:133) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD4-0-0] [lt=5] refresh tenant config(tenants=[], ret=-5019) [2024-09-26 16:00:38.958000] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.958050] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:38.973764] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:38.988419] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:38.988426] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:38.993329] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:38.993336] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:38.993354] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337638993326) [2024-09-26 16:00:38.993367] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337638892692, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:38.993387] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] get wrs ts(ls_id={id:1}, delta=223368229789, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:38.993403] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:38.993410] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:38.993417] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:38.993420] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:38.993426] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.993428] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:38.993430] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337638993373) [2024-09-26 16:00:38.995244] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337637995261]) [2024-09-26 16:00:38.995252] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337637995261], now0=[mts=1727337637995261], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:38.995258] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337637995097, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:38.995276] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:38.995294] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337637995097, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:38.995301] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337637995097, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:38.995320] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=18][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:38.995325] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:38.995330] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:38.995333] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:38.995336] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:38.995339] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:38.995353] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13] will sleep(sleep_us=1000, remain_us=27998979, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:38.997174] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:38.997182] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:38.997184] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:38.997186] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:38.997192] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:38.997233] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:38.997246] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:39.001069] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337638810733]}) [2024-09-26 16:00:39.005147] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500733, support_ssl=false) [2024-09-26 16:00:39.005171] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221500733, sess.client_addr_="127.0.0.1:36338") [2024-09-26 16:00:39.005177] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f9267004030, session_id:3221500733, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:39.005195] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:39.005206] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:39.005256] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9D-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:39.005425] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9D-0-0] [lt=7][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:39.005433] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9D-0-0] [lt=7][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"yLO(wA|p>$7'[vbg^#1l"}, ret=-4043) [2024-09-26 16:00:39.005439] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9D-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:39.005462] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9D-0-0] [lt=23][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:39.005470] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9D-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:39.005495] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9D-0-0] [lt=6] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500733, proxy_sessid:0}) [2024-09-26 16:00:39.005501] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9D-0-0] [lt=6] mark session id unused(sessid=3221500733) [2024-09-26 16:00:39.005506] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9D-0-0] [lt=3][errcode=0] server close connection(sessid=3221500733, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:39.005509] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9D-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500733, proxy_sessid=0) [2024-09-26 16:00:39.005511] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9D-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:39.005514] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9D-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500733, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:39.005537] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4015] close sql sock by user req(*s={this:0x7f9267004030, session_id:3221500733, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337639004971, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:39.005545] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] kill and revert session(conn.sessid_=3221500733, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:39.005548] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9267004030, session_id:3221500733, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337639004971, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:39.005553] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500733, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:39.009973] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:39.010102] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash time detail, (compute_wash_size_time=69, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:39.010738] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=912253, clean_num=31457) [2024-09-26 16:00:39.018688] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:39.018697] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.020723] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185906-0-0] [lt=24][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:39.020736] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185906-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:39.020758] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185906-0-0] [lt=21][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:39.020764] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185906-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:39.041417] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:39.041429] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:39.041433] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:39.041437] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:39.041440] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:39.041442] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:39.041445] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:39.041477] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=32][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:39.041480] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:39.041481] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:39.041483] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:39.041485] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:39.041487] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:39.041490] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:39.041497] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:39.041499] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:39.041501] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:39.041504] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:39.041506] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:39.041508] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:39.041511] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:39.041516] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:39.041521] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:39.041523] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:39.041524] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:39.041529] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:39.041532] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85880-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:39.048942] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:39.048951] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.048956] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 530.00) [2024-09-26 16:00:39.048962] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:39.048974] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:39.048984] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:39.051879] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=8] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:39.052169] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5A-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:39.052180] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5A-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:39.052201] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5A-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:39.079254] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:39.079263] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.084187] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=13] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:39.093657] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:39.093677] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337639093653) [2024-09-26 16:00:39.093683] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337638993373, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:39.093687] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D88E-0-0] [lt=13][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337639093671}) [2024-09-26 16:00:39.093703] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223368330118, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:39.093721] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:39.093729] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:39.093734] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:39.093737] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:39.093744] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.093746] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.093747] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337639093687) [2024-09-26 16:00:39.101264] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337638810733]}) [2024-09-26 16:00:39.109509] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:39.109518] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.139770] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21] success to notify_need_writing_throttling True [2024-09-26 16:00:39.139781] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.152463] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:39.152470] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-09-26 16:00:39.152832] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD3-0-0] [lt=16] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-09-26 16:00:39.152842] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD3-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:39.152870] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD3-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:39.163239] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=8][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337638810733]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:39.163253] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=13][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:39.172017] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:39.172026] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.193722] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:39.193735] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337639193720) [2024-09-26 16:00:39.193740] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337639093687, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:39.193765] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223368429956, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:39.193772] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:39.193781] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:39.193786] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:39.193795] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:39.193803] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.193808] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.193811] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337639193743) [2024-09-26 16:00:39.201302] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337638810733]}) [2024-09-26 16:00:39.202263] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27] success to notify_need_writing_throttling True [2024-09-26 16:00:39.202272] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.202498] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:39.202511] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:39.202518] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:39.211162] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:39.211304] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Wash time detail, (compute_wash_size_time=77, refresh_score_time=17, wash_time=1) [2024-09-26 16:00:39.211932] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=943710, clean_num=31457) [2024-09-26 16:00:39.232427] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:39.232437] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.239441] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588F-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:39.239482] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588F-0-0] [lt=40][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:39.239513] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648588F-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:39.262876] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:39.262884] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.293109] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:39.293120] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.294082] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:39.294095] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337639294078) [2024-09-26 16:00:39.294101] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337639193743, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:39.294127] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223368529620, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:39.294136] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:39.294147] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:39.294152] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:39.294156] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:39.294162] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.294165] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.294167] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337639294104) [2024-09-26 16:00:39.301950] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337638810733]}) [2024-09-26 16:00:39.302613] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337640231137, now=[mts=1727337638301450]) [2024-09-26 16:00:39.302621] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337640231137, now=[mts=1727337638301450], now0=[mts=1727337638301450], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:39.302628] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92691562e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337638301187, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:39.302644] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92691562e0}) [2024-09-26 16:00:39.302652] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92691562e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337638301187, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:39.302659] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92691562e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337638301187, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337638301423, use_das=false, nested_level=0, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92691562e0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:39.302676] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:39.302680] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:39.302683] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:39.302685] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:39.302687] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:39.302691] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:39.302695] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=998442, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337640301137) [2024-09-26 16:00:39.302698] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:39.302705] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:39.302707] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:39.302708] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:39.302714] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:39.302750] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:39.302753] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:39.311142] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.311154] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337639311139], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:39.311174] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337639311139], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:39.323421] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:39.323451] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.323786] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D3A28-0-0] [lt=9][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:39.339020] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337639339019, bytes: 14628460, bw: 0.003620 MB/s, add_ts: 1007000, add_bytes: 3822 [2024-09-26 16:00:39.352701] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:39.352753] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [2917][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10] start to check and clear obj lock(ls->get_ls_id()={id:1}) [2024-09-26 16:00:39.353668] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:39.353688] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.384930] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=9] current monitor number(seq_=-1) [2024-09-26 16:00:39.384970] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:00:39.384975] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.385006] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:39.394740] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:39.394752] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337639394737) [2024-09-26 16:00:39.394757] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337639294104, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:39.394763] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:39.394780] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:39.394784] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:39.395120] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574D-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:39.395134] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574D-0-0] [lt=12][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:39.395163] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574D-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:39.395196] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337639395196, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=428, wlock_time=18, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:39.395209] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:39.395225] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223368630167, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:39.395231] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:39.395237] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:39.395242] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:39.395245] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:39.395250] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.395253] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.395255] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337639395213) [2024-09-26 16:00:39.402541] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337639311139]}) [2024-09-26 16:00:39.411296] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [2940][T1_ArcFetcher][T1][YB42C0A8002D-0006230055E84B24-0-0] [lt=6] ObArchiveFetcher is running(thread_index=0) [2024-09-26 16:00:39.412356] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:39.412496] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=50, refresh_score_time=15, wash_time=12) [2024-09-26 16:00:39.413086] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=975167, clean_num=31457) [2024-09-26 16:00:39.416210] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25] success to notify_need_writing_throttling True [2024-09-26 16:00:39.416216] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.433759] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:156, tid:2932}]) [2024-09-26 16:00:39.446463] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:39.446470] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.454278] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.454298] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.454304] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.454310] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.454312] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.454314] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.455232] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.455239] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.455252] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.455258] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.455260] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.455262] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.456217] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.456228] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.456233] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.456237] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.456239] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.456241] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.456788] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.456794] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.456797] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.456800] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.456802] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.456803] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.458171] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.458177] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.458180] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.458183] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.458185] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.458186] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.458734] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.458740] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.458743] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.458746] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.458748] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.458749] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.459497] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.459504] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.459506] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.459509] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.459510] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.459512] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.460022] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.460027] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.460029] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.460032] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.460034] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.460035] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:39.461081] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.461093] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.461629] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.461647] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.463067] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.463076] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.464062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.464075] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.464977] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.464984] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.465502] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.465510] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.466968] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.466977] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.467735] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.467743] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.468969] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.468975] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.469895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.469903] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.470877] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.470886] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.471730] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.471738] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.472811] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.472819] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.473430] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.473437] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.474044] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.474052] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.474575] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.474582] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.475256] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.475264] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.476401] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.476416] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.476553] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:39.476560] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.476966] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.476973] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.477731] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.477737] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.478544] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.478550] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.479712] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.479722] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.480632] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.480639] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.481696] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.481708] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.482593] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.482602] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.483609] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.483621] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.484545] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.484554] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.485529] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] ====== tenant freeze timer task ====== [2024-09-26 16:00:39.485540] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:39.485602] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337635622676, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337635622633, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337635622724, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:39.485610] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.485622] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.485629] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:39.485686] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D8-0-0] [lt=2] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:39.485694] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D8-0-0] [lt=5] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:39.485698] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D8-0-0] [lt=2] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:39.485701] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D8-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:39.485705] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D8-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:39.485707] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D8-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:39.485709] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D8-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:39.485730] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D8-0-0] [lt=20] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:39.485732] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D8-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:39.485734] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D8-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:39.485737] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D8-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:39.485769] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851D8-0-0] [lt=2] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:39.486153] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.486164] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.487484] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.487490] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.488425] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.488434] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.489457] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.489463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.489972] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.490015] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.491398] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.491411] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.491930] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.491937] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.493325] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.493332] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.494300] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.494307] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.494825] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.494832] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.494853] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:39.494862] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337639494850) [2024-09-26 16:00:39.494867] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337639394760, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:39.494890] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223368730867, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:39.494899] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:39.494906] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:39.494911] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:39.494916] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:39.494921] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.494923] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.494925] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337639494869) [2024-09-26 16:00:39.495384] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.495392] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.496247] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:39.496257] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=7][errcode=-4038] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:39.496993] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:39.503100] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337639311139]}) [2024-09-26 16:00:39.507003] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:39.507012] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.521878] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185907-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:39.521899] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185907-0-0] [lt=20][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:39.521909] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185907-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:39.521914] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185907-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:39.534473] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4038, dropped:5, tid:2580}]) [2024-09-26 16:00:39.537275] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:39.537283] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.541163] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6413, mem_used=12478464) [2024-09-26 16:00:39.553113] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:39.562846] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=6] gc stale ls task succ [2024-09-26 16:00:39.568072] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:39.568083] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.594929] WDIAG [PALF] convert_to_ts (scn.cpp:265) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2024-09-26 16:00:39.594938] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1727337639594927, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1727337639594927, force_self_check=false, weak_read_refresh_interval=100000) [2024-09-26 16:00:39.594968] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:39.594980] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337639594966) [2024-09-26 16:00:39.594984] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337639494869, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:39.595009] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223368830703, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:39.595016] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:39.595024] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:39.595029] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:39.595031] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:39.595037] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.595039] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.595041] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337639594987) [2024-09-26 16:00:39.598441] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:39.598466] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.601752] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.601788] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.601796] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.601808] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.603313] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337639311139]}) [2024-09-26 16:00:39.613559] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:39.613662] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=53, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:39.614329] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1006624, clean_num=31457) [2024-09-26 16:00:39.622289] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855E3-0-0] [lt=10] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:39.629102] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:39.629120] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.658438] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:39.659363] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:39.659372] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.675036] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:39.689663] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14] success to notify_need_writing_throttling True [2024-09-26 16:00:39.689673] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.695523] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:39.695529] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:39.695533] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337639695520) [2024-09-26 16:00:39.695539] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337639594986, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:39.695552] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get_max_applied_scn(scn={val:1727114270763524000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:374, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:1727114270763524000, v:0}, max_applied_cb_scn_:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:39.695560] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get_min_unreplayed_log_info(lsn={lsn:629872191939}, scn={val:1727114270763524001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:00:39.695579] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=223368932020, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:39.695585] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:39.695591] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:39.695595] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:39.695597] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:39.695602] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.695604] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.695605] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337639695541) [2024-09-26 16:00:39.703334] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337639311139]}) [2024-09-26 16:00:39.719887] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:39.719896] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.734577] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=7] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:39.750201] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:39.750210] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.753146] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:39.757312] INFO [STORAGE.TRANS] print_statistics (ob_xa_define.cpp:467) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=6] xa statistics(total_active_xa_ctx_count=0, total_standby_clearup_count=0, total_success_xa_start=0, total_failure_xa_start=0, total_success_xa_prepare=0, total_failure_xa_prepare=0, total_success_xa_1pc_commit=0, total_failure_xa_1pc_commit=0, total_success_xa_2pc_commit=0, total_failure_xa_2pc_commit=0, total_failure_xa_rollback=0, total_success_dblink_promotion=0, total_failure_dblink_promotion=0, total_success_dblink=0, total_failure_dblink=0) [2024-09-26 16:00:39.757322] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=9] XA scheduler heartbeat task statistics(avg_time=12) [2024-09-26 16:00:39.777545] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337639777544, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007911, add_bytes: 0 [2024-09-26 16:00:39.780522] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=13] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-09-26 16:00:39.780525] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:39.780535] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-09-26 16:00:39.780532] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.780540] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=4] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=688) [2024-09-26 16:00:39.796142] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:39.796153] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337639796123) [2024-09-26 16:00:39.796158] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337639695541, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:39.796173] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223369032608, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:39.796178] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:39.796186] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:39.796191] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:39.796193] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:39.796198] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.796200] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.796202] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337639796161) [2024-09-26 16:00:39.803933] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337639311139]}) [2024-09-26 16:00:39.810782] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:39.810791] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.811867] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:39.811877] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337639811864], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:39.811899] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337639811864], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:39.811903] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:39.811906] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:39.815743] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:39.815845] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:39.816482] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1038081, clean_num=31457) [2024-09-26 16:00:39.821566] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=10] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.821568] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.821577] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.821581] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.821583] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.821590] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.821596] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.821606] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.821667] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.821676] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.821681] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:39.840997] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:39.841028] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.865820] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, table_name.ptr()="data_size:16, data:5F5F616C6C5F6D657267655F696E666F", ret=-5019) [2024-09-26 16:00:39.865832] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, ret=-5019) [2024-09-26 16:00:39.865838] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_merge_info, db_name=oceanbase) [2024-09-26 16:00:39.865841] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_merge_info) [2024-09-26 16:00:39.865845] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:39.865848] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:39.865851] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_merge_info' doesn't exist [2024-09-26 16:00:39.865853] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:39.865855] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:39.865857] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:39.865859] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:39.865861] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:39.865864] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:39.865866] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:39.865873] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:39.865875] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:39.865878] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:39.865880] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:39.865882] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_merge_info WHERE tenant_id = '1', ret=-5019) [2024-09-26 16:00:39.865884] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:39.865887] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:39.865892] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:39.865898] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:39.865901] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:39.865902] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:39.865908] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:39.865911] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:39.865914] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885082-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, aret=-5019, ret=-5019) [2024-09-26 16:00:39.865916] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:39.865919] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:39.865921] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:39.865923] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337639865753, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:39.865926] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:39.865929] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:49) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:39.865958] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:890) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] failed to load global merge info(ret=-5019, ret="OB_TABLE_NOT_EXIST", global_merge_info={tenant_id:1, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2024-09-26 16:00:39.865969] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1005) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] fail to refresh merge info(tmp_ret=-5019, tmp_ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:39.866209] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:39.866214] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:39.871332] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:39.871343] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.896736] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:39.896748] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337639896733) [2024-09-26 16:00:39.896753] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337639796161, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:39.896759] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:39.896768] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:39.896771] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:39.897112] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:39.897120] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:39.897124] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:39.897127] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:39.897130] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:39.897132] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:39.897135] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:39.897138] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:39.897139] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:39.897141] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:39.897143] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:39.897145] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:39.897147] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:39.897149] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:39.897164] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:39.897167] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:39.897169] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:39.897171] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:39.897173] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:39.897175] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:39.897177] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:39.897182] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:39.897186] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:39.897188] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:39.897189] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:39.897194] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:39.897197] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:39.897199] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574E-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:39.897202] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:39.897204] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:39.897206] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:39.897208] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337639897055, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:39.897211] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:39.897213] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:39.897237] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:39.897240] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337639897239, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=475, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:39.897246] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:39.897248] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:39.897264] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223369132244, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:39.897269] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:39.897275] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:39.897279] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:39.897281] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:39.897286] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.897289] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.897290] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337639897252) [2024-09-26 16:00:39.901591] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:39.901601] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.904526] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337639811864]}) [2024-09-26 16:00:39.917063] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] ====== checkpoint timer task ====== [2024-09-26 16:00:39.917485] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] succ to get rec scn(*this={this:0x7f92a6a04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1726077600526835000, v:0}, last_push_gc_task_ts_:1727334198745636, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:39.917496] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] tx ctx memtable get rec scn(this={ObITable:{this:0x7f92a22a4080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f92a22a4080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:39.917508] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1723174350926847000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-09-26 16:00:39.917523] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=4] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:39.917544] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}, min_rec_scn={val:1726163586617842000, v:0}, ls_id_={id:1}) [2024-09-26 16:00:39.917557] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:39.917559] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:39.917562] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:39.917567] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=1] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1726163586617842000, v:0}, checkpoint_scn_in_ls_meta={val:1726163586617842000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-09-26 16:00:39.917574] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}, curr_base_lsn={lsn:627966418944}, new_base_lsn={lsn:627966418944}, lsn={lsn:627966418944}) [2024-09-26 16:00:39.917579] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:627966418944}, ls->get_ls_id()={id:1}) [2024-09-26 16:00:39.917582] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-09-26 16:00:39.931883] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:39.931891] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.954718] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=723488, replace_num=15728) [2024-09-26 16:00:39.954730] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:39.962157] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14] success to notify_need_writing_throttling True [2024-09-26 16:00:39.962164] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.962177] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:39.970005] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [2708][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=187] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"2GB", log_disk_size:"2GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, mode:0, create_timestamp:1710828706488537, is_removed:false}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=15, log_id=1828900, offset=14449774}, ls_meta_entry:[10007](ver=0,mode=0,seq=609892), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:2}, create_status:1}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:49, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:79703013, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:1725, recv_lp_rpc_cnt:0, recv_mysql_cnt:1144, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:9708261, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:group_id = 2,queue_size = 0,recv_req_cnt = 7504,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334211521784 group_id = 10,queue_size = 0,recv_req_cnt = 3422,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334216415596 group_id = 9,queue_size = 0,recv_req_cnt = 7056,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334199387293 group_id = 1,queue_size = 0,recv_req_cnt = 4466,min_worker_cnt = 8,max_worker_cnt = 8,worker_cnt = 0,token_change = 1727336105580026 , rpc_stat_info: pcode=0x150a:cnt=20 pcode=0x150b:cnt=20 pcode=0x4a9:cnt=10 pcode=0x113:cnt=5, token_change_ts:1727334199363389, tenant_role:1}) [2024-09-26 16:00:39.970207] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [2708][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=202] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1727334198122806, is_removed:false}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:2}, create_status:1}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:12, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:2118255, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3235825, workers:12, nesting workers:0, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1727334198125585, tenant_role:0}) [2024-09-26 16:00:39.973883] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:39.980004] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-09-26 16:00:39.980012] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=6] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:39.980016] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=3] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:39.980019] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:39.981741] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] Dumping All Cache Objs(alloc_obj_list.count()=3, alloc_obj_list=[{obj_id:78, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157779}, {obj_id:79, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:5, added_to_lc:true, mem_used:175172}]) [2024-09-26 16:00:39.981751] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:39.992419] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21] success to notify_need_writing_throttling True [2024-09-26 16:00:39.992426] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:39.997318] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337638997232]) [2024-09-26 16:00:39.997336] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337638997232], now0=[mts=1727337638997232], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:39.997347] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:39.997343] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337638996245, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:39.997354] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:39.997361] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337639997344) [2024-09-26 16:00:39.997359] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:39.997365] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337639896756, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:39.997366] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337638996245, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:39.997380] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223369232895, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:39.997373] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337638996245, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:39.997386] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:39.997392] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=18][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:39.997393] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:39.997396] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:39.997397] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:39.997400] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:39.997400] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:39.997402] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:39.997405] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.997404] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:39.997407] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:39.997408] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:39.997409] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337639997368) [2024-09-26 16:00:39.997412] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=2000, remain_us=26996921, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:39.999571] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:39.999586] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:39.999590] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:39.999591] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:39.999596] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:39.999632] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:39.999638] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:40.004882] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337639811864]}) [2024-09-26 16:00:40.006260] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221500740, support_ssl=false) [2024-09-26 16:00:40.006284] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] sm conn init succ(conn.sessid_=3221500740, sess.client_addr_="127.0.0.1:36340") [2024-09-26 16:00:40.006289] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f92be35cf90, session_id:3221500740, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:40.006308] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:40.006316] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=2] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:40.006365] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9F-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:40.006557] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9F-0-0] [lt=15][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:40.006563] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9F-0-0] [lt=5][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"4`gd?PQ*xD)@EzW$I('-"}, ret=-4043) [2024-09-26 16:00:40.006569] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9F-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:40.006573] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9F-0-0] [lt=4][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:40.006581] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9F-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:40.006602] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9F-0-0] [lt=2] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500740, proxy_sessid:0}) [2024-09-26 16:00:40.006605] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9F-0-0] [lt=3] mark session id unused(sessid=3221500740) [2024-09-26 16:00:40.006609] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9F-0-0] [lt=2][errcode=0] server close connection(sessid=3221500740, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:40.006612] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9F-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500740, proxy_sessid=0) [2024-09-26 16:00:40.006614] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9F-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:40.006617] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584F9F-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500740, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:40.006684] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f92be35cf90, session_id:3221500740, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337640006103, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:40.006694] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] kill and revert session(conn.sessid_=3221500740, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:40.006696] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=2] can close safely, do destroy(*s={this:0x7f92be35cf90, session_id:3221500740, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337640006103, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:40.006700] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500740, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:40.016685] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:40.016796] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:40.017403] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1069538, clean_num=31457) [2024-09-26 16:00:40.021844] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185908-0-0] [lt=46][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:40.021853] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185908-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:40.021861] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185908-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:40.021869] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185908-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:40.022535] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:40.022542] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.042589] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:40.042601] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:40.042606] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:40.042609] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:40.042613] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:40.042615] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:40.042620] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:40.042641] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=20][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:40.042643] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:40.042645] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:40.042647] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:40.042650] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:40.042652] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:40.042654] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:40.042660] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:40.042663] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:40.042666] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:40.042667] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:40.042670] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:40.042672] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:40.042675] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:40.042682] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=6][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:40.042688] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:40.042690] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:40.042691] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:40.042697] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:40.042700] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:40.042703] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:40.042705] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:40.042708] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85881-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:40.042710] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85881-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:40.042731] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85881-0-0] [lt=20][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337640042518, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:40.042734] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85881-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:40.042740] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85881-0-0] [lt=5][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:40.042763] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85881-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:40.042767] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85881-0-0] [lt=4][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:40.052979] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=10] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:40.053274] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:40.053285] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:40.053301] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=15][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:40.053306] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=5][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:40.053310] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:40.053312] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:40.053315] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:40.053317] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:40.053318] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:40.053320] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:40.053321] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:40.053323] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:40.053327] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:40.053329] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:40.053335] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:40.053337] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:40.053339] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:40.053341] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:40.053343] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:40.053345] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:40.053347] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:40.053354] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:40.053359] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:40.053361] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:40.053362] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:40.053367] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:40.053370] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5C-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:40.053372] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2579][RSAsyncTask0][T0][YB42C0A8002D-0006230057584E5C-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:40.053377] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:40.053379] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:40.053381] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:40.053383] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337640053225, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:40.053387] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:40.053430] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:40.053504] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:40.053512] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.054966] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 12145.00) [2024-09-26 16:00:40.054974] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:40.054985] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:40.054991] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:40.084253] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=7] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:40.084271] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:40.084279] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.097912] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:40.097927] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337640097909) [2024-09-26 16:00:40.097932] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337639997367, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:40.097935] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D89A-0-0] [lt=9][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337640097922}) [2024-09-26 16:00:40.097948] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223369333487, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.097954] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:40.097961] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:40.097966] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:40.097968] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:40.097974] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.097976] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.097977] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337640097935) [2024-09-26 16:00:40.105742] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337639811864]}) [2024-09-26 16:00:40.114535] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:40.114547] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.144613] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:40.144624] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.155527] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:40.175067] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:40.175079] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.198093] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:40.198106] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337640198089) [2024-09-26 16:00:40.198111] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337640097935, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:40.198130] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223369434075, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.198136] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:40.198143] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:40.198148] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:40.198151] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:40.198165] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.198168] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.198169] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337640198114) [2024-09-26 16:00:40.198339] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=2] ====== check clog disk timer task ====== [2024-09-26 16:00:40.198347] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=7] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:40.198356] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:40.205343] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:40.205353] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.206004] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337639811864]}) [2024-09-26 16:00:40.218183] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:40.218307] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:40.218916] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1100995, clean_num=31457) [2024-09-26 16:00:40.237589] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:00:40.237598] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.240824] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:40.240835] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:40.240840] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:40.240843] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:40.240847] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:40.240849] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:40.240853] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:40.240855] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:40.240857] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:40.240859] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:40.240861] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:40.240863] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:40.240866] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:40.240868] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:40.240874] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:40.240876] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:40.240879] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:40.240881] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:40.240884] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:40.240886] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:40.240888] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:40.240894] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:40.240900] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:40.240901] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:40.240903] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:40.240909] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:40.240912] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:40.240915] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:40.240917] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:40.240919] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:40.240921] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:40.240923] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337640240755, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:40.240927] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:40.240929] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:40.240956] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:40.240958] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:40.240961] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=3][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:40.240963] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:40.240965] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:40.240968] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:40.240970] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485890-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:40.267917] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:40.267927] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.298282] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:40.298292] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.298310] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:40.298320] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337640298307) [2024-09-26 16:00:40.298325] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337640198114, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:40.298342] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223369534756, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.298347] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:40.298355] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:40.298360] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:40.298362] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:40.298368] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.298370] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.298371] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337640298328) [2024-09-26 16:00:40.303013] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337640231137, now=[mts=1727337639302749]) [2024-09-26 16:00:40.303021] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337640231137, now=[mts=1727337639302749], now0=[mts=1727337639302749], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:40.303027] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92691562e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337639302078, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:40.303047] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=19][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92691562e0}) [2024-09-26 16:00:40.303054] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92691562e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337639302078, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:40.303062] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92691562e0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337639302078, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337638301423, use_das=false, nested_level=0, session={this:0x7f926360e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92691562e0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:40.303080] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:40.303085] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:40.303088] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:40.303090] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:40.303092] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:40.303095] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:40.303099] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:40.303105] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:40.303106] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=1][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:40.303108] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:40.303112] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:40.303115] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2001692) [2024-09-26 16:00:40.303118] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:40.303133] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=13][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:40.303135] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:40.303137] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:40.303140] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:40.303145] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:40.303171] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:40.303173] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:40.303176] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:40.303178] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=1][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:40.303181] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:40.303184] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:40.303308] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7A-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:40.303318] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:40.303320] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:40.303346] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=17] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:40.303350] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:40.303656] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:40.303664] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=7][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:40.306913] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337639811864]}) [2024-09-26 16:00:40.312885] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.312893] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337640312882], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:40.312900] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337640312882], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:40.324529] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D3A88-0-0] [lt=13][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:40.328444] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:40.328466] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.339154] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1727337640339153, bytes: 14632000, bw: 0.003376 MB/s, add_ts: 1000134, add_bytes: 3540 [2024-09-26 16:00:40.355803] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:40.358717] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:40.358725] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.386034] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=9] current monitor number(seq_=-1) [2024-09-26 16:00:40.386049] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:40.389023] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24] success to notify_need_writing_throttling True [2024-09-26 16:00:40.389030] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.392223] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4E-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:40.392235] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4E-0-0] [lt=7] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:40.392241] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4E-0-0] [lt=5][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:40.392245] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4E-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:40.392248] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4E-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:40.392251] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4E-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:40.392253] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:40.392255] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:40.392259] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:40.392261] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4E-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.392283] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4E-0-0] [lt=0] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:40.399735] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:40.399745] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337640399717) [2024-09-26 16:00:40.399751] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337640298328, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:40.399759] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:40.399786] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:40.399792] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:40.400108] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574F-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:40.400119] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574F-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.400200] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788574F-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:40.400285] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337640400232, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=458, wlock_time=20, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:40.400331] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:40.400540] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1}, delta=223369636215, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.400553] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:40.400561] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:40.400595] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:40.400600] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:40.400611] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.400613] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.400615] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337640400338) [2024-09-26 16:00:40.407533] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337640312882]}) [2024-09-26 16:00:40.419252] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:40.419259] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.419443] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:40.419628] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=95, refresh_score_time=30, wash_time=2) [2024-09-26 16:00:40.420277] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1132452, clean_num=31457) [2024-09-26 16:00:40.449523] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:40.449530] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.455374] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.455375] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=25] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.455375] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.455383] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.455384] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.455388] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.455389] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.455400] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.455406] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.455925] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.455934] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.455938] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.456025] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.456046] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.456048] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.456067] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.456071] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.456074] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.457338] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.457336] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.457337] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.457341] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.457342] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.457344] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.457343] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.457345] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.457347] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.458298] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.458299] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.458299] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.458302] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.458302] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.458304] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.458304] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.458304] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.458307] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.459292] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.459296] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.459298] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.459282] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.459309] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.459302] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.459312] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.459314] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.459314] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=11][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.460262] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.460267] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.460266] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.460270] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:40.460265] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.461235] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.461329] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.461369] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.462222] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.462220] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.462225] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.462743] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.462789] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.462788] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.464167] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.464168] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.464168] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.465148] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.465182] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.465198] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.466153] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.466153] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.466193] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.467084] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.467098] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.467117] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.468131] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.468152] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.468152] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.469112] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.469138] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.469154] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.470079] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.470085] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.470114] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.471052] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.471063] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.471081] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.472040] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.472047] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.472073] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.472595] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.472595] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.472602] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.473107] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.473108] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.473170] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.473940] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.473972] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.473988] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.474911] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.474919] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.474920] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.475890] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.475891] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.475891] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.476892] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.476895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.476917] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.477452] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.477454] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.477492] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.478825] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.478830] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.478883] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.479801] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.479802] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.479813] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=42] success to notify_need_writing_throttling True [2024-09-26 16:00:40.479818] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.479850] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.480334] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.480335] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.480382] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.481773] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.481824] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.482174] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.482307] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.482402] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.482712] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.482872] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.482985] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.483708] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.483708] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.483713] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.484693] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.484698] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.484699] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.485664] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.485668] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:40.500311] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:40.500324] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337640500308) [2024-09-26 16:00:40.500329] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337640399756, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:40.500346] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223369736779, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.500352] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:40.500368] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:40.500394] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:40.500397] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:40.500411] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.500415] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.500416] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337640500332) [2024-09-26 16:00:40.507868] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337640312882]}) [2024-09-26 16:00:40.510095] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:40.510101] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.529771] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185909-0-0] [lt=56][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:40.529780] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185909-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:40.529789] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185909-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:40.529794] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185909-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:40.540414] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:40.540421] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.541340] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=1, size_used=6422, mem_used=12478464) [2024-09-26 16:00:40.549180] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [2925][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=12] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f92b77d02b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-09-26 16:00:40.556027] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:40.562910] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=9] gc stale ls task succ [2024-09-26 16:00:40.570642] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:40.570647] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.600952] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:40.600963] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:40.600969] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.600976] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.600997] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.601029] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:40.601051] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337640601026) [2024-09-26 16:00:40.601056] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337640500331, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:40.601072] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223369837437, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.601090] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:40.601097] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:40.601103] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:40.601105] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:40.601110] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.601112] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.601114] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337640601058) [2024-09-26 16:00:40.601546] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.601554] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.608746] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337640312882]}) [2024-09-26 16:00:40.621571] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:40.621686] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:40.622295] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1163909, clean_num=31457) [2024-09-26 16:00:40.625324] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] ====== traversal_flush timer task ====== [2024-09-26 16:00:40.625370] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d76030, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6163825430799493735, dag_cnt=1, dag_type_cnts=1) [2024-09-26 16:00:40.625378] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] success to create and add dag(ret=0, dag=0x7f9264d76030) [2024-09-26 16:00:40.625382] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1961) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(dag_cnt=1, map_size=1) [2024-09-26 16:00:40.625390] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1971) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(running_dag_net_map_size=0, blocking_dag_net_list_size=0) [2024-09-26 16:00:40.625381] INFO [STORAGE] flush (ob_tx_data_memtable.cpp:917) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] [TX DATA MERGE]schedule flush tx data memtable task done(ret=0, ret="OB_SUCCESS", get_ls_id()={id:1}, this=0x7f92a2294080, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:4, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337640625380, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:40.625393] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_COMPACTION_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=1, waiting_dag_count=0) [2024-09-26 16:00:40.625407] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_HIGH", low_limit=8, up_limit=8, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:40.625409] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_COMPACTION_MID", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:40.625411] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_HA_MID", low_limit=5, up_limit=5, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:40.625412] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(priority="PRIO_COMPACTION_LOW", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:40.625414] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_HA_LOW", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:40.625416] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(priority="PRIO_DDL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:40.625417] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(priority="PRIO_DDL_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:40.625419] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_TTL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:40.625420] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:40.625423] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=182) [2024-09-26 16:00:40.625425] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:40.625427] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:00:40.625428] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:40.625430] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:00:40.625431] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=1) [2024-09-26 16:00:40.625433] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=2) [2024-09-26 16:00:40.625438] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:40.625416] INFO [MDS] flush (mds_table_mgr.cpp:172) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=18] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:40.625449] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:00:40.625451] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:40.625453] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:00:40.625454] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:00:40.625468] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:00:40.625469] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:00:40.625471] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:00:40.625472] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:00:40.625474] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:00:40.625475] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:00:40.625477] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:00:40.625478] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625480] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625481] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625483] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625484] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625485] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625487] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625486] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=35] add dag success(dag=0x7f9264d76560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6659188001345271846, dag_cnt=2, dag_type_cnts=1) [2024-09-26 16:00:40.625488] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625490] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625490] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d76560) [2024-09-26 16:00:40.625491] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625493] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625494] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625496] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625497] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625493] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, create_flush_dag_time:1727337640625493, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452867, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200240}, checkpoint_unit_ptr:0x7f92a1a70850}, is_freeze_clock:true}) [2024-09-26 16:00:40.625501] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625503] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625505] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625506] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625508] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625509] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625511] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625512] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625501] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337640625479, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.625514] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625515] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625516] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625518] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625517] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=14] add dag success(dag=0x7f9264d76a90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-9069613569758001279, dag_cnt=3, dag_type_cnts=2) [2024-09-26 16:00:40.625519] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d76a90) [2024-09-26 16:00:40.625519] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625521] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625522] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:40.625521] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, create_flush_dag_time:1727337640625521, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452892, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200489}, checkpoint_unit_ptr:0x7f92a1a70080}, is_freeze_clock:true}) [2024-09-26 16:00:40.625524] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625525] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-09-26 16:00:40.625527] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-09-26 16:00:40.625528] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-09-26 16:00:40.625530] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-09-26 16:00:40.625525] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337640625515, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.625531] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:40.625533] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:40.625534] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:40.625534] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d76fc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1288986994123435108, dag_cnt=4, dag_type_cnts=3) [2024-09-26 16:00:40.625536] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d76fc0) [2024-09-26 16:00:40.625536] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:40.625537] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:40.625539] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:40.625537] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337640625537, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452911, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847222}, checkpoint_unit_ptr:0x7f92a1a71020}, is_freeze_clock:true}) [2024-09-26 16:00:40.625540] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:40.625542] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:40.625543] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:40.625544] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:40.625546] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:40.625541] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337640625532, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.625547] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:40.625549] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:40.625550] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d774f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5355160120942094702, dag_cnt=5, dag_type_cnts=4) [2024-09-26 16:00:40.625551] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d774f0) [2024-09-26 16:00:40.625554] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:40.625553] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337640625553, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452929, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847223}, checkpoint_unit_ptr:0x7f92a1a717f0}, is_freeze_clock:true}) [2024-09-26 16:00:40.625556] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:40.625557] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:40.625559] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:40.625560] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:40.625556] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337640625548, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.625562] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:40.625564] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625565] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:40.625566] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625568] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:40.625569] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625571] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:40.625565] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405042) [2024-09-26 16:00:40.625575] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625577] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:40.625577] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:40.625579] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625580] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:40.625582] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625583] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:40.625585] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625580] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405070) [2024-09-26 16:00:40.625586] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:40.625587] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:40.625588] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625589] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:40.625591] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625592] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:40.625594] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:40.625590] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405087) [2024-09-26 16:00:40.625595] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0) [2024-09-26 16:00:40.625596] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:40.625596] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, scheduled_task_count=0) [2024-09-26 16:00:40.625598] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0) [2024-09-26 16:00:40.625600] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, scheduled_task_count=0) [2024-09-26 16:00:40.625601] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0) [2024-09-26 16:00:40.625603] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-09-26 16:00:40.625599] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405104) [2024-09-26 16:00:40.625604] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0) [2024-09-26 16:00:40.625606] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-09-26 16:00:40.625606] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:40.625607] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0) [2024-09-26 16:00:40.625609] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, scheduled_task_count=0) [2024-09-26 16:00:40.625610] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_net_count=0) [2024-09-26 16:00:40.625611] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] add dag success(dag=0x7f9264d77a20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1994541446894784994, dag_cnt=6, dag_type_cnts=5) [2024-09-26 16:00:40.625612] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_net_count=0) [2024-09-26 16:00:40.625612] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d77a20) [2024-09-26 16:00:40.625613] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_net_count=0) [2024-09-26 16:00:40.625615] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_net_count=0) [2024-09-26 16:00:40.625616] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_net_count=0) [2024-09-26 16:00:40.625617] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_net_count=0) [2024-09-26 16:00:40.625619] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_net_count=0) [2024-09-26 16:00:40.625621] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_net_count=0) [2024-09-26 16:00:40.625622] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1991) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=184) [2024-09-26 16:00:40.625614] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, create_flush_dag_time:1727337640625614, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209556884, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:226}, checkpoint_unit_ptr:0x7f92a1a73f00}, is_freeze_clock:true}) [2024-09-26 16:00:40.625630] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337640625626, task_id:YB42C0A8002D-0006230056D8C644-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.625636] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=5] schedule one task(task=0x7f9264d763f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.625635] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=20] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337640625608, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.625643] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337640625642, task_id:YB42C0A8002D-0006230056D8C645-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.625645] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=2] schedule one task(task=0x7f9264d76920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.625650] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640625650, task_id:YB42C0A8002D-0006230056D8C646-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.625655] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=4] schedule one task(task=0x7f9264d76e50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.625660] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337640625659, task_id:YB42C0A8002D-0006230056D8C647-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.625662] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=2] schedule one task(task=0x7f9264d77380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.625667] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640625666, task_id:YB42C0A8002D-0006230056D8C648-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.625669] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=2] schedule one task(task=0x7f9264d778b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.625672] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640625672, task_id:YB42C0A8002D-0006230056D8C649-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.625674] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=2] schedule one task(task=0x7f9264d77de0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.625703] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.625708] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.625710] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:40.625710] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.625714] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.625717] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726157244792838000) [2024-09-26 16:00:40.625717] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.625719] INFO [STORAGE] ready_for_flush (ob_tx_data_memtable.cpp:865) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=1] memtable is frozen yet.(this=0x7f92a2294080) [2024-09-26 16:00:40.625745] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.625748] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.625749] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:40.625752] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.625757] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847222}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:40.625775] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.625804] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.625807] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.625808] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.625810] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.625811] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.625812] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:40.625815] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.625817] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:00:40.625818] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.625822] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241777443612000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.625841] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.625843] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.625844] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:40.625847] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.625823] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726164780922097000, old_tablet_snaphsot_version=1726164781026522000, tablet={this:0x7f92b863df20, is_inited:true, wash_score:1727337640625574000, hold_ref_cnt:true, ref_cnt:2, version:2, length:487, tablet_addr:[816017-1849-0-2305843626616512512-2](offset=1269760,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, data_tablet_id:{id:200240}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726164781225658000, v:0}, clog_checkpoint_scn:{val:1726164781225658000, v:0}, ddl_checkpoint_scn:{val:1711609700160391001, v:0}, snapshot_version:1726164781026522000, multi_version_start:1726164780922097000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726164780922097000, cur_report_version:1726164780922097000, data_checksum:2501178096, row_count:147053}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:1711609700101236001, v:0}, ddl_snapshot_version:1711609700074566000, max_sync_storage_schema_version:1711609699705840, max_serialized_medium_scn:0, ddl_execution_id:0, ddl_data_format_version:17180000769, ddl_commit_scn:{val:1711609700160391001, v:0}, mds_checkpoint_scn:{val:1726201924319685000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1711609699705840}, table_store_addr:{ptr:{this:0x7f92b863e5d8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844241870487552-0](offset=1179648,size=1148,type=2,seq=0), type:"MAJOR", tablet_id:{id:1152921504606847223}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726164780922097000, v:0}}, snapshot_version:1726164780922097000, max_merge_version:1726164780922097000}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337640625548, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843251880595456-2](offset=1249280,size=226,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276217058-0](offset=1249506,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255101821304-0](offset=1249656,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:1726164780922097000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1711609699719192000, v:0}, create_commit_version:1711609699719192000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a717f0, 0x7f92a1a72f60, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.625884] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=60] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:40.625891] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.625975] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d77f50, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8791616776033068452, dag_cnt=7, dag_type_cnts=6) [2024-09-26 16:00:40.625981] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] success to create and add dag(ret=0, dag=0x7f9264d77f50) [2024-09-26 16:00:40.625985] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337640625984, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405141, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847408}, checkpoint_unit_ptr:0x7f92a1a746d0}, is_freeze_clock:true}) [2024-09-26 16:00:40.625990] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337640625643, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626026] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=10] add dag success(dag=0x7f9264d78480, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6550230011160095889, dag_cnt=8, dag_type_cnts=7) [2024-09-26 16:00:40.626030] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d78480) [2024-09-26 16:00:40.626031] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337640626031, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405158, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847409}, checkpoint_unit_ptr:0x7f92a1a75670}, is_freeze_clock:true}) [2024-09-26 16:00:40.626035] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337640626000, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626045] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d789b0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6809498982591245614, dag_cnt=9, dag_type_cnts=8) [2024-09-26 16:00:40.626047] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d789b0) [2024-09-26 16:00:40.626048] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, create_flush_dag_time:1727337640626048, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405174, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:200004}, checkpoint_unit_ptr:0x7f92a1a74ea0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626051] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337640626042, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626060] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d78ee0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6815944285528637599, dag_cnt=10, dag_type_cnts=9) [2024-09-26 16:00:40.626062] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d78ee0) [2024-09-26 16:00:40.626063] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, create_flush_dag_time:1727337640626063, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209506214, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1}, checkpoint_unit_ptr:0x7f92a1a76610}, is_freeze_clock:true}) [2024-09-26 16:00:40.626066] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337640626058, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626078] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264d79410, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-4182959058374342994, dag_cnt=11, dag_type_cnts=10) [2024-09-26 16:00:40.626080] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d79410) [2024-09-26 16:00:40.626082] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, create_flush_dag_time:1727337640626082, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405204, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:412}, checkpoint_unit_ptr:0x7f92a1a75e40}, is_freeze_clock:true}) [2024-09-26 16:00:40.626085] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337640626075, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626093] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d79940, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1586518267935340, dag_cnt=12, dag_type_cnts=11) [2024-09-26 16:00:40.626095] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d79940) [2024-09-26 16:00:40.626097] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, create_flush_dag_time:1727337640626096, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405220, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:444}, checkpoint_unit_ptr:0x7f92a1a775b0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626100] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337640626091, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626136] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=32] add dag success(dag=0x7f9264d79e70, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3044367941103776476, dag_cnt=13, dag_type_cnts=12) [2024-09-26 16:00:40.626139] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d79e70) [2024-09-26 16:00:40.626140] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, create_flush_dag_time:1727337640626140, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455964, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:309}, checkpoint_unit_ptr:0x7f92a1a76de0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626143] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337640626132, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626152] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7a3a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1596830851104282047, dag_cnt=14, dag_type_cnts=13) [2024-09-26 16:00:40.626154] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7a3a0) [2024-09-26 16:00:40.626156] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, create_flush_dag_time:1727337640626156, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455986, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:331}, checkpoint_unit_ptr:0x7f92a1a77d80}, is_freeze_clock:true}) [2024-09-26 16:00:40.626161] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337640626150, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626170] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7a8d0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2360195962081676673, dag_cnt=15, dag_type_cnts=14) [2024-09-26 16:00:40.626172] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7a8d0) [2024-09-26 16:00:40.626174] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, create_flush_dag_time:1727337640626174, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405262, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:458}, checkpoint_unit_ptr:0x7f92a1a7a490}, is_freeze_clock:true}) [2024-09-26 16:00:40.626177] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337640626168, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626189] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264d7ae00, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5675615614797020460, dag_cnt=16, dag_type_cnts=15) [2024-09-26 16:00:40.626191] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7ae00) [2024-09-26 16:00:40.626193] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, create_flush_dag_time:1727337640626192, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405279, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:487}, checkpoint_unit_ptr:0x7f92a1a79cc0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626196] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337640626186, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626207] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d7b330, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-612435342908088006, dag_cnt=17, dag_type_cnts=16) [2024-09-26 16:00:40.626209] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7b330) [2024-09-26 16:00:40.626210] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, create_flush_dag_time:1727337640626210, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405296, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:455}, checkpoint_unit_ptr:0x7f92a1a794f0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626213] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337640626204, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626221] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7b860, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7868818205990873505, dag_cnt=18, dag_type_cnts=17) [2024-09-26 16:00:40.626223] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7b860) [2024-09-26 16:00:40.626247] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=23] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, create_flush_dag_time:1727337640626246, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405313, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:492}, checkpoint_unit_ptr:0x7f92a1a78550}, is_freeze_clock:true}) [2024-09-26 16:00:40.626250] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337640626220, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626260] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7bd90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7071731214611448811, dag_cnt=19, dag_type_cnts=18) [2024-09-26 16:00:40.626298] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=38] success to create and add dag(ret=0, dag=0x7f9264d7bd90) [2024-09-26 16:00:40.626300] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, create_flush_dag_time:1727337640626300, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405347, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:501}, checkpoint_unit_ptr:0x7f92a1a78d20}, is_freeze_clock:true}) [2024-09-26 16:00:40.626306] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337640626257, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626316] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7c2c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8731750006000903454, dag_cnt=20, dag_type_cnts=19) [2024-09-26 16:00:40.626318] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7c2c0) [2024-09-26 16:00:40.626320] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, create_flush_dag_time:1727337640626320, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405365, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:472}, checkpoint_unit_ptr:0x7f92a1a7ac60}, is_freeze_clock:true}) [2024-09-26 16:00:40.626323] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337640626313, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626333] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7c7f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6766388540859538791, dag_cnt=21, dag_type_cnts=20) [2024-09-26 16:00:40.626337] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] success to create and add dag(ret=0, dag=0x7f9264d7c7f0) [2024-09-26 16:00:40.626339] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, create_flush_dag_time:1727337640626339, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405384, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:342}, checkpoint_unit_ptr:0x7f92a1a7cba0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626342] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337640626330, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626351] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7cd20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2986199386776543176, dag_cnt=22, dag_type_cnts=21) [2024-09-26 16:00:40.626353] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7cd20) [2024-09-26 16:00:40.626354] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337640626354, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405418, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:140}, checkpoint_unit_ptr:0x7f92a1a7b430}, is_freeze_clock:true}) [2024-09-26 16:00:40.626380] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=25] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337640626349, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626390] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7d250, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6947063304478012270, dag_cnt=23, dag_type_cnts=22) [2024-09-26 16:00:40.626392] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7d250) [2024-09-26 16:00:40.626393] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337640626393, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405436, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101009}, checkpoint_unit_ptr:0x7f92a1a7bc00}, is_freeze_clock:true}) [2024-09-26 16:00:40.626397] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337640626387, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626405] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7d780, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5843378789243662202, dag_cnt=24, dag_type_cnts=23) [2024-09-26 16:00:40.626407] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7d780) [2024-09-26 16:00:40.626408] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337640626408, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405453, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101010}, checkpoint_unit_ptr:0x7f92a1a7c3d0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626411] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337640626403, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626420] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7dcb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1841823684513746176, dag_cnt=25, dag_type_cnts=24) [2024-09-26 16:00:40.626421] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d7dcb0) [2024-09-26 16:00:40.626423] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, create_flush_dag_time:1727337640626423, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405470, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:323}, checkpoint_unit_ptr:0x7f92a1a7d370}, is_freeze_clock:true}) [2024-09-26 16:00:40.626426] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337640626418, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626438] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264d7e1e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-2678407261265885087, dag_cnt=26, dag_type_cnts=25) [2024-09-26 16:00:40.626440] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7e1e0) [2024-09-26 16:00:40.626442] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, create_flush_dag_time:1727337640626442, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405487, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:377}, checkpoint_unit_ptr:0x7f92a1a7e310}, is_freeze_clock:true}) [2024-09-26 16:00:40.626445] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337640626435, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626457] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d7e710, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7175089551615529855, dag_cnt=27, dag_type_cnts=26) [2024-09-26 16:00:40.626476] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=18] success to create and add dag(ret=0, dag=0x7f9264d7e710) [2024-09-26 16:00:40.626478] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, create_flush_dag_time:1727337640626478, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405503, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:297}, checkpoint_unit_ptr:0x7f92a1a7eae0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626481] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337640626453, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626492] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7ec40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3730526293242960203, dag_cnt=28, dag_type_cnts=27) [2024-09-26 16:00:40.626494] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7ec40) [2024-09-26 16:00:40.626519] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=22] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337640626518, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405519, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101065}, checkpoint_unit_ptr:0x7f92a1a7db40}, is_freeze_clock:true}) [2024-09-26 16:00:40.626522] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337640626488, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626533] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7f170, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=4663792628280492528, dag_cnt=29, dag_type_cnts=28) [2024-09-26 16:00:40.626536] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7f170) [2024-09-26 16:00:40.626537] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337640626537, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405536, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:333}, checkpoint_unit_ptr:0x7f92a3bc4850}, is_freeze_clock:true}) [2024-09-26 16:00:40.626542] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337640626529, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626553] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7f6a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8399557287176188414, dag_cnt=30, dag_type_cnts=29) [2024-09-26 16:00:40.626555] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d7f6a0) [2024-09-26 16:00:40.626557] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337640626557, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405552, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101066}, checkpoint_unit_ptr:0x7f92a3bc5020}, is_freeze_clock:true}) [2024-09-26 16:00:40.626560] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337640626549, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626569] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7fbd0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8068938909778324349, dag_cnt=31, dag_type_cnts=30) [2024-09-26 16:00:40.626571] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7fbd0) [2024-09-26 16:00:40.626576] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, create_flush_dag_time:1727337640626576, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405569, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:334}, checkpoint_unit_ptr:0x7f92a3bc57f0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626579] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337640626566, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626589] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d80100, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7665372330263152188, dag_cnt=32, dag_type_cnts=31) [2024-09-26 16:00:40.626591] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d80100) [2024-09-26 16:00:40.626593] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, create_flush_dag_time:1727337640626592, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405586, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:451}, checkpoint_unit_ptr:0x7f92a3bc4080}, is_freeze_clock:true}) [2024-09-26 16:00:40.626595] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337640626586, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626627] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=28] add dag success(dag=0x7f9264d80630, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6759548225246904317, dag_cnt=33, dag_type_cnts=32) [2024-09-26 16:00:40.626629] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d80630) [2024-09-26 16:00:40.626631] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, create_flush_dag_time:1727337640626631, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405602, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:378}, checkpoint_unit_ptr:0x7f92a1a7f2b0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626634] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337640626623, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626651] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=13] add dag success(dag=0x7f9264d80b60, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8944614441376604790, dag_cnt=34, dag_type_cnts=33) [2024-09-26 16:00:40.626654] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d80b60) [2024-09-26 16:00:40.626656] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, create_flush_dag_time:1727337640626656, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209456003, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:121}, checkpoint_unit_ptr:0x7f92a3bc5fc0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626660] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337640626647, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626672] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264d81090, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=507863347806783454, dag_cnt=35, dag_type_cnts=34) [2024-09-26 16:00:40.626675] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d81090) [2024-09-26 16:00:40.626677] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, create_flush_dag_time:1727337640626677, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405633, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:215}, checkpoint_unit_ptr:0x7f92a3bc86d0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626680] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337640626669, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626689] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d815c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1604282355669434429, dag_cnt=36, dag_type_cnts=35) [2024-09-26 16:00:40.626737] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=47] success to create and add dag(ret=0, dag=0x7f9264d815c0) [2024-09-26 16:00:40.626742] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, create_flush_dag_time:1727337640626742, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405649, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:456}, checkpoint_unit_ptr:0x7f92a3bc6790}, is_freeze_clock:true}) [2024-09-26 16:00:40.626746] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337640626686, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626765] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d81af0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3553649416163308074, dag_cnt=37, dag_type_cnts=36) [2024-09-26 16:00:40.626768] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d81af0) [2024-09-26 16:00:40.626770] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337640626770, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405666, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:154}, checkpoint_unit_ptr:0x7f92a3bc6f60}, is_freeze_clock:true}) [2024-09-26 16:00:40.626774] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337640626753, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626786] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d820a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8097406237500329300, dag_cnt=38, dag_type_cnts=37) [2024-09-26 16:00:40.626789] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d820a0) [2024-09-26 16:00:40.626790] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337640626790, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405682, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101015}, checkpoint_unit_ptr:0x7f92a3bc7730}, is_freeze_clock:true}) [2024-09-26 16:00:40.626793] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337640626783, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626803] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d825d0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3976350754940852513, dag_cnt=39, dag_type_cnts=38) [2024-09-26 16:00:40.626832] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=29] success to create and add dag(ret=0, dag=0x7f9264d825d0) [2024-09-26 16:00:40.626837] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337640626837, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405699, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101016}, checkpoint_unit_ptr:0x7f92a3bc7f00}, is_freeze_clock:true}) [2024-09-26 16:00:40.626867] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=30] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337640626800, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626903] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=10] add dag success(dag=0x7f9264d82b00, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8720304653345806796, dag_cnt=40, dag_type_cnts=39) [2024-09-26 16:00:40.626906] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d82b00) [2024-09-26 16:00:40.626908] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, create_flush_dag_time:1727337640626908, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405716, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101063}, checkpoint_unit_ptr:0x7f92a3bca610}, is_freeze_clock:true}) [2024-09-26 16:00:40.626924] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=16] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337640626878, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626935] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d83030, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1830562711513853292, dag_cnt=41, dag_type_cnts=40) [2024-09-26 16:00:40.626937] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d83030) [2024-09-26 16:00:40.626939] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, create_flush_dag_time:1727337640626939, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405733, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:452}, checkpoint_unit_ptr:0x7f92a3bc9e40}, is_freeze_clock:true}) [2024-09-26 16:00:40.626942] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337640626931, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626954] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264d83560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1536604874905918360, dag_cnt=42, dag_type_cnts=41) [2024-09-26 16:00:40.626957] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d83560) [2024-09-26 16:00:40.626958] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337640626958, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405750, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:328}, checkpoint_unit_ptr:0x7f92a3bc8ea0}, is_freeze_clock:true}) [2024-09-26 16:00:40.626961] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337640626951, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.626970] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d83a90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6933628521298738662, dag_cnt=43, dag_type_cnts=42) [2024-09-26 16:00:40.626972] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d83a90) [2024-09-26 16:00:40.626974] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337640626974, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405767, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:330}, checkpoint_unit_ptr:0x7f92a3bc9670}, is_freeze_clock:true}) [2024-09-26 16:00:40.626997] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=23] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337640626968, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.627009] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d83fc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1056199536334423564, dag_cnt=44, dag_type_cnts=43) [2024-09-26 16:00:40.627013] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d83fc0) [2024-09-26 16:00:40.627016] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337640627016, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405784, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:329}, checkpoint_unit_ptr:0x7f92a3bcb5b0}, is_freeze_clock:true}) [2024-09-26 16:00:40.627019] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337640627004, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.627028] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d844f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7405463607918801432, dag_cnt=45, dag_type_cnts=44) [2024-09-26 16:00:40.627030] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d844f0) [2024-09-26 16:00:40.627032] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, create_flush_dag_time:1727337640627032, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405801, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:332}, checkpoint_unit_ptr:0x7f92a3bcade0}, is_freeze_clock:true}) [2024-09-26 16:00:40.627037] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337640627025, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.627056] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=15] add dag success(dag=0x7f9264d84a20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6699723950958743903, dag_cnt=46, dag_type_cnts=45) [2024-09-26 16:00:40.627080] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=24] success to create and add dag(ret=0, dag=0x7f9264d84a20) [2024-09-26 16:00:40.627085] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, create_flush_dag_time:1727337640627085, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405818, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101067}, checkpoint_unit_ptr:0x7f92a3bcbd80}, is_freeze_clock:true}) [2024-09-26 16:00:40.627089] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337640627052, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.627096] INFO [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:655) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] traversal_flush successfully(ls_->get_ls_id()={id:1}, flush_tasks=[{table:0x7f92a1a70850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a70080, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71020, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a717f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71fc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72790, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72f60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73730, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73f00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a746d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75670, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a74ea0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76610, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75e40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a775b0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76de0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a77d80, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7a490, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a79cc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a794f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78550, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78d20, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7ac60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7cba0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7b430, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7bc00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7c3d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7d370, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7e310, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7eae0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7db40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a3bc4850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}]) [2024-09-26 16:00:40.627169] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=70] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-09-26 16:00:40.632185] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:40.632193] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.636571] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1, storage_schema:{this:0x7f92684ab2b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.636593] INFO [STORAGE] prepare_tx_data_list (ob_tx_data_memtable.cpp:266) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=20] construct list more than once. skip this time(this=0x7f92a2294080) [2024-09-26 16:00:40.636597] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, old_min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726159999077775000, old_tablet_snaphsot_version=1726163995122514000, tablet={this:0x7f92b8637b60, is_inited:true, wash_score:1727337640624381000, hold_ref_cnt:true, ref_cnt:2, version:2, length:503, tablet_addr:[814679-1621-0-2305843643796488192-2](offset=1376256,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:200240}, data_tablet_id:{id:200240}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726158221127945000, v:0}, clog_checkpoint_scn:{val:1726163995200406000, v:0}, ddl_checkpoint_scn:{val:1711609699643766000, v:0}, snapshot_version:1726163995122514000, multi_version_start:1726159999077775000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726158220122116000, cur_report_version:1726158220122116000, data_checksum:2675562562, row_count:146957}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:1711609699490656001, v:0}, ddl_snapshot_version:1711609699456885000, max_sync_storage_schema_version:1711609699311648, max_serialized_medium_scn:0, ddl_execution_id:1, ddl_data_format_version:17180000769, ddl_commit_scn:{val:1711609699643766000, v:0}, mds_checkpoint_scn:{val:1726163586417012001, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1711609699311648}, table_store_addr:{ptr:{this:0x7f92b86381f4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814679-1621-0-2305845774100242432-0](offset=1351680,size=2575,type=2,seq=0), type:"MAJOR", tablet_id:{id:200240}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726158220122116000, v:0}}, snapshot_version:1726158220122116000, max_merge_version:1726158220122116000}}, minor_tables:{{i:0, addr:[814679-1621-0-2305844370719681039-0](offset=1354255,size=1268,type=2,seq=0), type:"MINI", tablet_id:{id:200240}, scn_range:{start_scn:{val:1726158221127945000, v:0}, end_scn:{val:1726159107115548000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726159106450605000, upper_trans_version:1726159106450605000}, {i:1, addr:[814679-1621-0-2305846187490848515-0](offset=1355523,size=2960,type=2,seq=0), type:"MINOR", tablet_id:{id:200240}, scn_range:{start_scn:{val:1726159107115548000, v:0}, end_scn:{val:1726163187167245000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726163187058024000, upper_trans_version:1726163187058024000}, {i:2, addr:[814679-1621-0-2305844371793427091-0](offset=1358483,size=1269,type=2,seq=0), type:"MINI", tablet_id:{id:200240}, scn_range:{start_scn:{val:1726163187167245000, v:0}, end_scn:{val:1726163995200406000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726163995191699000, upper_trans_version:1726163995191699000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337640625479, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814679-1621-0-2305843593330610176-2](offset=1363968,size=544,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814679-1621-0-2305843163833881120-0](offset=1364512,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814679-1621-0-2305843255101936304-0](offset=1364656,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814679-1621-0-2305843235774583701-0](offset=1364885,size=211,type=2,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:1726158220122116000}, medium_info_list:{ptr:(nil), addr:[814679-1621-0-2305843463407857664-0](offset=1372160,size=423,type=2,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1711609699370224000, v:0}, create_commit_version:1711609699370224000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a70850, 0x7f92a1a71fc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.636653] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726158220122116000) [2024-09-26 16:00:40.636663] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.639170] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=6] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.639188] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=17] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337640625532, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:40.639195] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.639316] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=29] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:40.639324] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=7] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.639335] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847222}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:40.639447] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.639488] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=40] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337640625548, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:40.639477] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=117] succeed to build merge ctx(tablet_id={id:1152921504606847222}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847222}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640625743, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=13.59ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.639496] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92685e9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.639599] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:40.639604] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.639611] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92685e9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847223}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:40.639656] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=37] succeed to build merge ctx(tablet_id={id:1152921504606847223}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92685e9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847223}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640625803, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=13.80ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.639801] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=30] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:40.639944] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=22] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:40.640584] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=5] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=8366, total_rows=58562) [2024-09-26 16:00:40.640669] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=4] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=7999, total_rows=55993) [2024-09-26 16:00:40.640717] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:40.640733] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=8] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.640754] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337640625608, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.640767] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752755184, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828752755184, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.640783] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.640787] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.640794] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:226}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752755184, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.640814] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:226}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752755184, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:226}, table_count:1, [{i:0, table_key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}, merge_scn:{val:1726241777443612000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640625793, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=14.99ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.640840] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C649-0-0, dag_ret:0, dag_status:2, start_time:1727337640625674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}, scanned_row_cnt_arr:0x7f92685e7480, output_block_cnt_arr:0x7f92685e7488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:56623104, latest_update_ts:1727337640625793, estimated_finish_time:1727337657640867}) [2024-09-26 16:00:40.640849] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264d77de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C649-0-0, dag_ret:0, dag_status:2, start_time:1727337640625674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:40.640856] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=6] task finish process(ret=0, start_time=1727337640625792, end_time=1727337640640855, runtime=15063, *this={this:0x7f9264d77de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C649-0-0, dag_ret:0, dag_status:2, start_time:1727337640625674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:40.641027] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=2] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.641048] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C647-0-0, dag_ret:0, dag_status:2, start_time:1727337640625662, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad470, output_block_cnt_arr:0x7f92684ad478, concurrent_cnt:1, estimate_row_cnt:329211, estimate_occupy_size:150994944, latest_update_ts:1727337640625743, estimated_finish_time:1727337657641119}) [2024-09-26 16:00:40.641061] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=12] succeed to init merge ctx(task={this:0x7f9264d77380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C647-0-0, dag_ret:0, dag_status:2, start_time:1727337640625662, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.641067] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=6] task finish process(ret=0, start_time=1727337640625742, end_time=1727337640641067, runtime=15325, *this={this:0x7f9264d77380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C647-0-0, dag_ret:0, dag_status:2, start_time:1727337640625662, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.641100] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=7] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C648-0-0, dag_ret:0, dag_status:2, start_time:1727337640625669, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685e9470, output_block_cnt_arr:0x7f92685e9478, concurrent_cnt:1, estimate_row_cnt:332829, estimate_occupy_size:150994944, latest_update_ts:1727337640625803, estimated_finish_time:1727337657641171}) [2024-09-26 16:00:40.641121] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=11] succeed to init merge ctx(task={this:0x7f9264d778b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C648-0-0, dag_ret:0, dag_status:2, start_time:1727337640625669, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.641129] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=8] task finish process(ret=0, start_time=1727337640625801, end_time=1727337640641129, runtime=15328, *this={this:0x7f9264d778b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C648-0-0, dag_ret:0, dag_status:2, start_time:1727337640625669, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.641641] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=7] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.641649] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=7] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.641729] INFO [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:329) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=3] pre-process commit versions row(get_ls_id()={id:1}, do_recycle_=true, current_time=1727337640641728, prev_recycle_time=0) [2024-09-26 16:00:40.641756] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aad52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a36060, count:4, allocator:0x7f9269f787d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a36350, local_datums:0x7f9263a36350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:40.641773] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c76060, count:4, allocator:0x7f92864427d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9262c76350, local_datums:0x7f9262c76350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:40.641784] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=24] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aad52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:4, allocator:0x7f9269f79558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3e350, local_datums:0x7f9263a3e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:40.641794] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=16] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684e8060, count:4, allocator:0x7f9286443558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684e8350, local_datums:0x7f92684e8350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:40.642048] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:40.642065] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=7][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:40.642122] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.642122] ERROR issue_dba_error (ob_log.cpp:1875) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=9][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.642128] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.642128] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.642137] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=9][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:40.642137] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=9][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:40.642141] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=4][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:40.642140] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:40.642143] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f92665b0140, hex: 3230323430393131303133333438, cstr: 20240911013348},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f92665b014e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b00e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b0118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3080, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7221, data_checksum:4284205003, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f926846c030, buf_size:7221, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1004563202, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:00:40.642143] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=3][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9264d5c140, hex: 3230323430393131303133333131, cstr: 20240911013311},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9264d5c14e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c0e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-15038, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7227, data_checksum:1452123993, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92729ee030, buf_size:7227, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1818264584, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:00:40.642160] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=16][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:40.642160] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=17][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:40.642162] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:40.642162] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:40.642164] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:40.642164] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:40.642166] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:40.642169] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=5][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:40.642168] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a36350, local_datums:0x7f9263a36350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a363d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a36408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a36350, local_datums:0x7f9263a36350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a363d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a36408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9269f78118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269f78118, limit_param:null, stmt_allocator:0x7f9269f787d8, allocator:0x7f9269f787d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:-1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a36350, local_datums:0x7f9263a36350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a363d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a36408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:00:40.642171] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9262c76350, local_datums:0x7f9262c76350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c763d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9262c76350, local_datums:0x7f9262c76350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c763d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9286442118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286442118, limit_param:null, stmt_allocator:0x7f92864427d8, allocator:0x7f92864427d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9262c76350, local_datums:0x7f9262c76350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c763d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:00:40.642229] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=57][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:40.642229] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=61][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:40.642231] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9262c76350, local_datums:0x7f9262c76350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c763d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9286442118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286442118, limit_param:null, stmt_allocator:0x7f92864427d8, allocator:0x7f92864427d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9262c76350, local_datums:0x7f9262c76350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c763d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:00:40.642231] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a36350, local_datums:0x7f9263a36350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a363d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a36408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9269f78118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269f78118, limit_param:null, stmt_allocator:0x7f9269f787d8, allocator:0x7f9269f787d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:-1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a36350, local_datums:0x7f9263a36350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a363d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a36408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:00:40.642264] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=32][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:40.642264] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=32][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:40.642267] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=1] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.642266] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=1] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.642281] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=10] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337640625548, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.642281] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=10] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337640625532, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.642291] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:40.642291] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:40.642305] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:40.642305] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:40.642309] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:40.642309] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:40.642313] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=3] task finish process(ret=-4184, start_time=1727337640641682, end_time=1727337640642312, runtime=630, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C648-0-0, dag_ret:0, dag_status:2, start_time:1727337640625669, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.642321] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C648-0-0, dag_ret:0, dag_status:2, start_time:1727337640625669, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:40.642311] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337640641715, end_time=1727337640642311, runtime=596, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C647-0-0, dag_ret:0, dag_status:2, start_time:1727337640625662, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.642330] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=19][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C647-0-0, dag_ret:0, dag_status:2, start_time:1727337640625662, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:40.642331] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=16661, dag_cnt=45, dag_cnts_[dag.get_type()]=44, &dag=0x7f9264d774f0, dag={ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C648-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640625669, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.642341] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C648-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337640625666, task_id:YB42C0A8002D-0006230056D8C648-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.642365] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=6] dag finished(dag_ret=-4184, runtime=16703, dag_cnt=44, dag_cnts_[dag.get_type()]=43, &dag=0x7f9264d76fc0, dag={ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C647-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640625662, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.642377] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C647-0-0] [lt=9] succeed to del sys task(removed_task={start_time:1727337640625659, task_id:YB42C0A8002D-0006230056D8C647-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.642422] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=226, range=0x7f92aaf52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:5, allocator:0x7f92c183a7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:40.642610] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640642606, task_id:YB42C0A8002D-0006230056D8C64A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.642617] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=6] schedule one task(task=0x7f9264d78310, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.642626] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337640642625, task_id:YB42C0A8002D-0006230056D8C64B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.642629] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=3] schedule one task(task=0x7f9264d78840, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.642665] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.642671] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.642674] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.642681] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.643972] INFO [STORAGE] get_ls_min_end_scn (ob_ls_tablet_service.cpp:5618) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=9] get ls min end scn finish(ls_checkpoint={val:1726163586617842000, v:0}) [2024-09-26 16:00:40.643983] WDIAG [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:761) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=7][errcode=-4023] failed to get snapshot version(ret=-4023, MTL_ID()=1) [2024-09-26 16:00:40.643988] INFO [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:773) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=4] get tx data recycle scn finish.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, recycle_scn={val:1726076389100953000, v:0}, min_end_scn_from_old_tablets={val:4611686018427387903, v:0}, min_end_scn_from_latest_tablets={val:1726076389100953000, v:0}) [2024-09-26 16:00:40.643994] WDIAG [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:336) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=5][errcode=-4023] get recycle ts failed.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:40.643999] WDIAG [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:236) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=2][errcode=-4023] process commit version row failed.(ret=-4023, ret="OB_EAGAIN", this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337640625380, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:40.644011] INFO [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:254) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=12] [TX DATA MERGE]pre process for merge done(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}, this=0x7f92a2294080, tg=time guard have no click events for optimization, fake_tx_data_guard={tx_data:{tx_id:{txid:9223372036854775807}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337640625380, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:40.644025] WDIAG [STORAGE.COMPACTION] pre_process_tx_data_table_merge_ (ob_tx_table_merge_task.cpp:117) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=12][errcode=-4023] do pre process for tx data table merge failed.(ret=-4023, ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, table={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337640625380, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:40.644033] WDIAG [STORAGE.COMPACTION] inner_init_ctx (ob_tx_table_merge_task.cpp:157) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=8][errcode=-4023] pre process tx data table for merge failed.(ret=-4023, ret="OB_EAGAIN", ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.644040] WDIAG [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1061) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=6][errcode=-4023] fail to inner init ctx(ret=-4023, tablet_id={id:49402}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, create_snapshot_version:0, is_full_merge:false, merge_level:1, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:4, range_array:[], concurrent_cnt:0, is_inited:false}, schema_ctx:{schema_version:1, storage_schema:{this:0x7f92684ab2b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:49402}, table_count:1, [{i:0, table_key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref:4}]}, schedule_major:false, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}, merge_scn:{val:4611686018427387903, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:NULL, compaction_filter:NULL, time_guard:total=18.33ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}) [2024-09-26 16:00:40.644062] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:975) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=22][errcode=-4023] failed to build merge ctx(ret=-4023, ctx->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.644065] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:994) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=3][errcode=-4023] sstable merge finish(ret=-4023, ctx=0x7f92684aa060, task={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C644-0-0, dag_ret:0, dag_status:2, start_time:1727337640625635, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:40.644072] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=7][errcode=-4023] failed to process task(ret=-4023) [2024-09-26 16:00:40.644074] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=2] task finish process(ret=-4023, start_time=1727337640625702, end_time=1727337640644074, runtime=18372, *this={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C644-0-0, dag_ret:0, dag_status:2, start_time:1727337640625635, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:40.644081] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=6][errcode=-4023] failed to do work(ret=-4023, *task_={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C644-0-0, dag_ret:0, dag_status:2, start_time:1727337640625635, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:40.644087] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=4] dag finished(dag_ret=-4023, runtime=18452, dag_cnt=43, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264d76030, dag={ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C644-0-0, dag_ret:-4023, dag_status:5, start_time:1727337640625635, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}) [2024-09-26 16:00:40.644095] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2869][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C644-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337640625626, task_id:YB42C0A8002D-0006230056D8C644-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.644108] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640644105, task_id:YB42C0A8002D-0006230056D8C64C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.644122] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=13] schedule one task(task=0x7f9264d78d70, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.644130] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.644134] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.644136] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.644143] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.644146] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.644151] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.644153] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.644166] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=10] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.644171] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.644183] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.644947] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.644961] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=14] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337640625479, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=18, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:40.644989] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.645309] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:40.645368] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=37] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.645435] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=61] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200240}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.645497] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=55] succeed to build merge ctx(tablet_id={id:200240}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200240}, table_count:2, [{i:0, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8637b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640625698, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=19.73ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.645875] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=45] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:40.646474] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=6] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C645-0-0, dag_ret:0, dag_status:2, start_time:1727337640625645, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d460, output_block_cnt_arr:0x7f9264b1d468, concurrent_cnt:1, estimate_row_cnt:85417, estimate_occupy_size:134217728, latest_update_ts:1727337640625698, estimated_finish_time:1727337657646538}) [2024-09-26 16:00:40.646487] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=12] succeed to init merge ctx(task={this:0x7f9264d76920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C645-0-0, dag_ret:0, dag_status:2, start_time:1727337640625645, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.646495] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=8] task finish process(ret=0, start_time=1727337640625694, end_time=1727337640646495, runtime=20801, *this={this:0x7f9264d76920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C645-0-0, dag_ret:0, dag_status:2, start_time:1727337640625645, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.646531] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=3] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.646601] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:3, allocator:0x7f92636927d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:40.646621] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=18] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:3, allocator:0x7f9263693558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5e320, local_datums:0x7f9263b5e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:40.646874] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=16][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:40.646950] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.646954] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf2041c8 0xf217c49 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.646984] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=30][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:40.646987] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:40.646989] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9264d60108, hex: 313030303030333830, cstr: 100000380},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d600a8, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d600e0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-29413, column_count:20, rowkey_column_count:3, has_column_checksum:0, row_count:100, row_store_type:0, opt:0, var_column_count:0, row_offset:16082, original_length:16422, max_merged_trans_version:1726189672100745000, data_length:16454, data_zlength:5272, data_checksum:4171874375, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9263bde030, buf_size:5272, data_size:16454, row_count:100, column_count:20, max_merged_trans_version:1726189672100745000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:371002229, row_count_delta:22, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16422}) [2024-09-26 16:00:40.647024] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=34][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:40.647026] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:40.647027] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:40.647029] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:40.647030] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1526) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, partition_fuser_->get_result_row()={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b58148, local_datums:0x7f9263b58148}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b58190, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b581c8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b583c0, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b583f8, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, fuse_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9263692118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263692118, limit_param:null, stmt_allocator:0x7f92636927d8, allocator:0x7f92636927d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5e320, local_datums:0x7f9263b5e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9263692e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263692e98, limit_param:null, stmt_allocator:0x7f9263693558, allocator:0x7f9263693558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320133, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5e320, local_datums:0x7f9263b5e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337640625479, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:40.647221] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=191][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9263692118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263692118, limit_param:null, stmt_allocator:0x7f92636927d8, allocator:0x7f92636927d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5e320, local_datums:0x7f9263b5e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9263692e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263692e98, limit_param:null, stmt_allocator:0x7f9263693558, allocator:0x7f9263693558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320133, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5e320, local_datums:0x7f9263b5e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337640625479, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:40.647336] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=114][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:40.647338] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=1] partition merge iter row count(i=0, row_count=23, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.647384] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=41] partition merge iter row count(i=1, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337640625479, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.647394] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:40.647407] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=3][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:40.647413] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=6][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:40.647442] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=28] task finish process(ret=-4184, start_time=1727337640646545, end_time=1727337640647441, runtime=896, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C645-0-0, dag_ret:0, dag_status:2, start_time:1727337640625645, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.647490] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=48][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C645-0-0, dag_ret:0, dag_status:2, start_time:1727337640625645, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:40.647500] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=6] dag finished(dag_ret=-4184, runtime=21855, dag_cnt=42, dag_cnts_[dag.get_type()]=42, &dag=0x7f9264d76560, dag={ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C645-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640625645, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.647509] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C645-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337640625642, task_id:YB42C0A8002D-0006230056D8C645-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.647530] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337640647526, task_id:YB42C0A8002D-0006230056D8C64D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.647536] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=6] schedule one task(task=0x7f9264d792a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.647593] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.647597] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.647600] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.647604] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.647608] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861eec0, is_inited:true, wash_score:1727337640625574000, hold_ref_cnt:true, ref_cnt:2, version:2, length:493, tablet_addr:[816017-1849-0-2305843633059586048-2](offset=1892352,size=581,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1}, data_tablet_id:{id:1}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:778798285, row_count:7249}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861f59c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844357835309056-0](offset=1884160,size=1256,type=2,seq=0), type:"MAJOR", tablet_id:{id:1}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077591663360001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843964845802728-0](offset=1885416,size=890,type=2,seq=0), type:"MINOR", tablet_id:{id:1}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178578410947000, upper_trans_version:1726178578410947000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337640626058, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcd4f0, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241621541056000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405180}, this:0x7f92a3bcd4f0, timestamp:1727334209405180, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:53, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277807268534000, v:0}, rec_scn:{val:1726241631569302000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771609600-2](offset=1888256,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843169203114317-0](offset=1888589,size=149,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881234914-0](offset=1888738,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828751995331000, v:0}, create_commit_version:1710828751995331000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a76610, 0x7f92a3bcd4f0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.647670] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.647680] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241621541056000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.647883] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:40.647899] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=9] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.648769] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337640626042, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.648785] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828761605424, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828761605424, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.648775] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8640e00, is_inited:true, wash_score:1727337640625574000, hold_ref_cnt:true, ref_cnt:2, version:2, length:506, tablet_addr:[816017-1849-0-2305843647017144320-1](offset=806912,size=594,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, data_tablet_id:{id:200489}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725991206240736000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1725991205062061000, cur_report_version:1725991205062061000, data_checksum:1029137533, row_count:46137450}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1725120000848360, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1725120000848360}, table_store_addr:{ptr:null, addr:[816017-1849-0-2305845308095737856-1](offset=798720,size=2141,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718219357-0](offset=800861,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255101372665-0](offset=801017,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1725991205062061000}, medium_info_list:{ptr:(nil), addr:[816017-1849-0-2305843505283219456-0](offset=802816,size=462,type=2,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1725120001569133000, v:0}, create_commit_version:1725120001569133000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a75670, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:40.648798] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=23] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:40.648800] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.648804] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.648805] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.648811] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200004}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1710828761605424, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.648840] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=23] succeed to build merge ctx(tablet_id={id:200004}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828761605424, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200004}, table_count:1, [{i:0, table_key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8621da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640644141, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.67ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.648870] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=22] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64C-0-0, dag_ret:0, dag_status:2, start_time:1727337640644122, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab460, output_block_cnt_arr:0x7f92684ab468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337640644141, estimated_finish_time:1727337657648873}) [2024-09-26 16:00:40.648877] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d78d70, type:15, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64C-0-0, dag_ret:0, dag_status:2, start_time:1727337640644122, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.648884] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=6] task finish process(ret=0, start_time=1727337640644138, end_time=1727337640648884, runtime=4746, *this={this:0x7f9264d78d70, type:15, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64C-0-0, dag_ret:0, dag_status:2, start_time:1727337640644122, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.648907] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.648940] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=32] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337640625515, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=17, max_schema_version_on_memtable=1725120002010696) [2024-09-26 16:00:40.648947] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=17, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.649002] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=12][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:40.649040] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=6] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1887, total_rows=92463) [2024-09-26 16:00:40.649043] ERROR issue_dba_error (ob_log.cpp:1875) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.649044] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.649046] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.649053] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:40.649056] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:40.649049] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200489}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:19, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.649057] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e20a8, hex: 0000000000000000, int: 0},idx=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92684e20e0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2118, hex: 58A3C56A4A280BE8, int: -1726241732082425000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-19590, column_count:10, rowkey_column_count:5, has_column_checksum:0, row_count:224, row_store_type:0, opt:0, var_column_count:0, row_offset:15527, original_length:16363, max_merged_trans_version:1726241777443612000, data_length:16363, data_zlength:16363, data_checksum:2057046089, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b7004090, buf_size:16363, data_size:16363, row_count:224, column_count:10, max_merged_trans_version:1726241777443612000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1006468742, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16363}) [2024-09-26 16:00:40.649069] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:40.649071] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:40.649073] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:40.649074] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:40.649076] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:40.649077] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:40.649065] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:200489}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200489}, table_count:2, [{i:0, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640625838, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=23.21ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.649079] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b43e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b43e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f92c183a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c183a118, limit_param:null, stmt_allocator:0x7f92c183a7d8, allocator:0x7f92c183a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:94314390480696, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b43e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337640625608, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}) [2024-09-26 16:00:40.649134] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=55][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:40.649135] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b43e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f92c183a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c183a118, limit_param:null, stmt_allocator:0x7f92c183a7d8, allocator:0x7f92c183a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:94314390480696, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b43e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337640625608, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}]) [2024-09-26 16:00:40.649169] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=34][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:40.649172] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=3] partition merge iter row count(i=0, row_count=224, ghost_row_count=0, pkey={tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, table={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337640625608, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.649185] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:40.649194] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:40.649197] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:40.649199] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337640642378, end_time=1727337640649199, runtime=6821, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C649-0-0, dag_ret:0, dag_status:2, start_time:1727337640625674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:40.649207] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C649-0-0, dag_ret:0, dag_status:2, start_time:1727337640625674, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:40.649215] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=23540, dag_cnt=41, dag_cnts_[dag.get_type()]=41, &dag=0x7f9264d77a20, dag={ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C649-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640625674, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}) [2024-09-26 16:00:40.649222] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C649-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640625672, task_id:YB42C0A8002D-0006230056D8C649-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.649364] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C646-0-0, dag_ret:0, dag_status:2, start_time:1727337640625654, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685eb460, output_block_cnt_arr:0x7f92685eb468, concurrent_cnt:1, estimate_row_cnt:254187, estimate_occupy_size:138412032, latest_update_ts:1727337640625838, estimated_finish_time:1727337657649429}) [2024-09-26 16:00:40.649377] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f9264d76e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C646-0-0, dag_ret:0, dag_status:2, start_time:1727337640625654, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.649383] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=6] task finish process(ret=0, start_time=1727337640625836, end_time=1727337640649383, runtime=23547, *this={this:0x7f9264d76e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C646-0-0, dag_ret:0, dag_status:2, start_time:1727337640625654, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.649394] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=3] schedule one task(task=0x7f9262d14030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.649424] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=28] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.649431] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337640649429, task_id:YB42C0A8002D-0006230056D8C64E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.649436] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=4] schedule one task(task=0x7f9264d797d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.649514] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a5e060, count:3, allocator:0x7f92864e07d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a5e320, local_datums:0x7f9264a5e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:40.649534] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=17] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d2e060, count:3, allocator:0x7f92864e1558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:40.649544] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.649550] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.649552] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.649557] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.649561] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.649568] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.649677] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=200004, range=0x7f92aa952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665aa060, count:3, allocator:0x7f92b33d87d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92665aa320, local_datums:0x7f92665aa320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:40.649718] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:40.649762] ERROR issue_dba_error (ob_log.cpp:1875) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.649765] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.649773] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:40.649775] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:40.649777] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a74070, hex: 9F094D0300000000, int: 55380383},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a740a8, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a740e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3044, column_count:19, rowkey_column_count:3, has_column_checksum:0, row_count:83, row_store_type:0, opt:5, var_column_count:0, row_offset:16109, original_length:16381, max_merged_trans_version:1726163995623997000, data_length:16381, data_zlength:7759, data_checksum:3416849541, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265a76030, buf_size:7759, data_size:16381, row_count:83, column_count:19, max_merged_trans_version:1726163995623997000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3537734793, row_count_delta:83, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16381}) [2024-09-26 16:00:40.649802] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=24][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:40.649804] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:40.649805] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:40.649807] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:40.649851] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=24][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:40.649808] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d2e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d2e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d2e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d2e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f92864e0e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92864e0e98, limit_param:null, stmt_allocator:0x7f92864e1558, allocator:0x7f92864e1558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:-1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d2e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d2e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337640625515, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}) [2024-09-26 16:00:40.649871] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=63][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:40.649891] ERROR issue_dba_error (ob_log.cpp:1875) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.649894] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.649901] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:40.649903] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:40.649872] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d2e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d2e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f92864e0e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92864e0e98, limit_param:null, stmt_allocator:0x7f92864e1558, allocator:0x7f92864e1558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:-1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9287d2e320, local_datums:0x7f9287d2e320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d2e4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d2e6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337640625515, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}]) [2024-09-26 16:00:40.649910] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=38][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:40.649905] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e6070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e60a8, hex: C852A9F488570BE8, int: -1726189786434547000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e60e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19071, column_count:8, rowkey_column_count:3, has_column_checksum:0, row_count:218, row_store_type:0, opt:0, var_column_count:0, row_offset:15549, original_length:16361, max_merged_trans_version:1726190661653507000, data_length:16361, data_zlength:5996, data_checksum:2604580500, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9268592030, buf_size:5996, data_size:16361, row_count:218, column_count:8, max_merged_trans_version:1726190661653507000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:83488362, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16361}) [2024-09-26 16:00:40.649915] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=10][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:40.649919] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=4][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:40.649912] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.649920] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:40.649922] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:40.649925] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=9] partition merge iter row count(i=1, row_count=83, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, table={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337640625515, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.649934] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:40.649945] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:40.649949] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:40.649951] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337640649451, end_time=1727337640649951, runtime=500, *this={this:0x7f9262d14030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C646-0-0, dag_ret:0, dag_status:2, start_time:1727337640625654, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.649970] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=18][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262d14030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C646-0-0, dag_ret:0, dag_status:2, start_time:1727337640625654, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:40.649978] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=24324, dag_cnt=40, dag_cnts_[dag.get_type()]=40, &dag=0x7f9264d76a90, dag={ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C646-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640625654, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.649986] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C646-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640625650, task_id:YB42C0A8002D-0006230056D8C646-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.649923] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92665aa320, local_datums:0x7f92665aa320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92665aa320, local_datums:0x7f92665aa320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f92b33d8118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b33d8118, limit_param:null, stmt_allocator:0x7f92b33d87d8, allocator:0x7f92b33d87d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92665aa320, local_datums:0x7f92665aa320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337640626042, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:40.649999] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=75][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:40.650001] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92665aa320, local_datums:0x7f92665aa320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f92b33d8118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b33d8118, limit_param:null, stmt_allocator:0x7f92b33d87d8, allocator:0x7f92b33d87d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92665aa320, local_datums:0x7f92665aa320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92665aa480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337640626042, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:40.650047] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=46][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:40.650051] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=3] partition merge iter row count(i=0, row_count=218, ghost_row_count=0, pkey={tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337640626042, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.650063] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:40.650074] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:40.650077] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:40.650079] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337640649635, end_time=1727337640650079, runtime=444, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64C-0-0, dag_ret:0, dag_status:2, start_time:1727337640644122, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.650087] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64C-0-0, dag_ret:0, dag_status:2, start_time:1727337640644122, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:40.650095] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=5972, dag_cnt=39, dag_cnts_[dag.get_type()]=39, &dag=0x7f9264d789b0, dag={ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640644122, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.650102] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64C-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640644105, task_id:YB42C0A8002D-0006230056D8C64C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.650281] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640650278, task_id:YB42C0A8002D-0006230056D8C64F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.650286] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=4] schedule one task(task=0x7f9264d79d00, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.650293] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337640650292, task_id:YB42C0A8002D-0006230056D8C650-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.650296] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=3] schedule one task(task=0x7f9264d7a230, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.650330] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.650334] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.650336] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.650340] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.650344] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8632d40, is_inited:true, wash_score:1727337640625574000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[816017-1849-0-2305843650239590400-2](offset=2027520,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:444}, data_tablet_id:{id:444}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1509715702, row_count:4}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8633440, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844406153818112-0](offset=2011136,size=1301,type=2,seq=0), type:"MAJOR", tablet_id:{id:444}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077198228842001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843949813544213-0](offset=2012437,size=876,type=2,seq=0), type:"MINOR", tablet_id:{id:444}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726177998310996000, upper_trans_version:1726177998310996000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337640626091, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bce490, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209461185}, this:0x7f92a3bce490, timestamp:1727334209461185, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:54, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277298911712000, v:0}, rec_scn:{val:1726241718316746000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697998848-2](offset=2019328,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161889100-0](offset=2019660,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881365998-0](offset=2019822,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447886032-0](offset=2020048,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a775b0, 0x7f92a3bce490, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.650400] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.650407] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.650431] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.650434] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.650435] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.650438] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.650441] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8631da0, is_inited:true, wash_score:1727337640625574000, hold_ref_cnt:true, ref_cnt:2, version:2, length:515, tablet_addr:[816017-1849-0-2305843656681984000-2](offset=1970176,size=603,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:309}, data_tablet_id:{id:309}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:810117565, row_count:8}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86324c4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844526412836864-0](offset=1945600,size=1413,type=2,seq=0), type:"MAJOR", tablet_id:{id:309}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077442448649001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843990615668101-0](offset=1947013,size=914,type=2,seq=0), type:"MINOR", tablet_id:{id:309}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178572306341000, upper_trans_version:1726178572306341000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337640626132, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcdcc0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241316899724000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405581}, this:0x7f92a3bcdcc0, timestamp:1727334209405581, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:51, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277651138194000, v:0}, rec_scn:{val:1726241616921722000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697937408-2](offset=1957888,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843189604278604-0](offset=1958220,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881304564-0](offset=1958388,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447824598-0](offset=1958614,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a76de0, 0x7f92a3bcdcc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.650488] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=47] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.650493] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241316899724000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.650597] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337640626058, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.650613] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=14] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:0, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.650627] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.650630] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.650637] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:0, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.650656] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:1}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:0, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1}, table_count:1, [{i:0, table_key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}, merge_scn:{val:1726241621541056000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640647587, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.05ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.650690] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=26] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64D-0-0, dag_ret:0, dag_status:2, start_time:1727337640647536, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}, scanned_row_cnt_arr:0x7f9263a45480, output_block_cnt_arr:0x7f9263a45488, concurrent_cnt:1, estimate_row_cnt:3, estimate_occupy_size:6291456, latest_update_ts:1727337640647587, estimated_finish_time:1727337657650692}) [2024-09-26 16:00:40.650699] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264d792a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64D-0-0, dag_ret:0, dag_status:2, start_time:1727337640647536, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:40.650705] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=5] task finish process(ret=0, start_time=1727337640647585, end_time=1727337640650704, runtime=3119, *this={this:0x7f9264d792a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64D-0-0, dag_ret:0, dag_status:2, start_time:1727337640647536, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:40.650731] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=2] schedule one task(task=0x7f9272926030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.650833] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=1, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:8}, cols_desc:{ObIArray:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b2060, count:5, allocator:0x7f926919e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b2360, local_datums:0x7f92684b2360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:40.651251] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=16][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:40.651292] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.651296] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.651302] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:40.651304] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:40.651306] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 17, flag: 0, null: 0, ptr: 0x7f9262c6e178, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e0a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 15, flag: 0, null: 0, ptr: 0x7f9262c6e189, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e118, hex: 60D2640CCC290BE8, int: -1726240075808452000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-20032, column_count:8, rowkey_column_count:5, has_column_checksum:0, row_count:158, row_store_type:0, opt:0, var_column_count:0, row_offset:15849, original_length:16421, max_merged_trans_version:1726241621541056000, data_length:16421, data_zlength:16421, data_checksum:3599888316, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926f204090, buf_size:16421, data_size:16421, row_count:158, column_count:8, max_merged_trans_version:1726241621541056000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1714289658, row_count_delta:2, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16421}) [2024-09-26 16:00:40.651317] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=11][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:40.651319] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:40.651321] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:40.651322] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:40.651323] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:40.651325] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:40.651326] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b2360, local_datums:0x7f92684b2360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b23a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b2360, local_datums:0x7f92684b2360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b23a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f926919e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926919e118, limit_param:null, stmt_allocator:0x7f926919e7d8, allocator:0x7f926919e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b2360, local_datums:0x7f92684b2360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b23a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337640626058, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:40.651383] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=56][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:40.651385] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b2360, local_datums:0x7f92684b2360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b23a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f926919e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926919e118, limit_param:null, stmt_allocator:0x7f926919e7d8, allocator:0x7f926919e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b2360, local_datums:0x7f92684b2360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b23a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b2450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337640626058, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:40.651421] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=36][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:40.651423] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=1] partition merge iter row count(i=0, row_count=158, ghost_row_count=0, pkey={tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, table={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337640626058, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.651449] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=22][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:40.651470] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:40.651474] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:40.651475] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337640650799, end_time=1727337640651475, runtime=676, *this={this:0x7f9272926030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64D-0-0, dag_ret:0, dag_status:2, start_time:1727337640647536, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:40.651482] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9272926030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64D-0-0, dag_ret:0, dag_status:2, start_time:1727337640647536, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:40.651489] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=3953, dag_cnt=38, dag_cnts_[dag.get_type()]=38, &dag=0x7f9264d78ee0, dag={ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640647536, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}) [2024-09-26 16:00:40.651497] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64D-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337640647526, task_id:YB42C0A8002D-0006230056D8C64D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.651563] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337640651560, task_id:YB42C0A8002D-0006230056D8C651-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.651570] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=6] schedule one task(task=0x7f9264d7a760, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.651607] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.651611] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.651613] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.651618] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.651621] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8633ce0, is_inited:true, wash_score:1727337640625574000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[816017-1849-0-2305843644870795264-2](offset=1941504,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:331}, data_tablet_id:{id:331}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1433772493, row_count:436}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86343bc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844404006256640-0](offset=1933312,size=1299,type=2,seq=0), type:"MAJOR", tablet_id:{id:331}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726076731692125000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843945518499091-0](offset=1934611,size=872,type=2,seq=0), type:"MINOR", tablet_id:{id:331}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178440167442000, upper_trans_version:1726178440167442000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337640626150, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd32b0, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241465393532000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209503876}, this:0x7f92a3bd32b0, timestamp:1727334209503876, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:64, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277479152983000, v:0}, rec_scn:{val:1726242365923499000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843367845400576-2](offset=1937408,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719356238-0](offset=1937742,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881284074-0](offset=1937898,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447804108-0](offset=1938124,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a77d80, 0x7f92a3bd32b0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.651677] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.651683] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241465393532000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.651926] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337640626000, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.651939] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=11] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:40.651942] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9263a4b480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.652149] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3223, total_rows=22561) [2024-09-26 16:00:40.652156] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=6] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.652163] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847409}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.652182] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:1152921504606847409}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9263a4b480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847409}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8640e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640642654, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=9.51ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.652390] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64B-0-0, dag_ret:0, dag_status:2, start_time:1727337640642629, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b5d0, output_block_cnt_arr:0x7f9263a4b5d8, concurrent_cnt:1, estimate_row_cnt:32439, estimate_occupy_size:18874368, latest_update_ts:1727337640642654, estimated_finish_time:1727337657652398}) [2024-09-26 16:00:40.652401] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=11] succeed to init merge ctx(task={this:0x7f9264d78840, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64B-0-0, dag_ret:0, dag_status:2, start_time:1727337640642629, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.652407] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=6] task finish process(ret=0, start_time=1727337640642651, end_time=1727337640652407, runtime=9756, *this={this:0x7f9264d78840, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64B-0-0, dag_ret:0, dag_status:2, start_time:1727337640642629, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.652431] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=3] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.652528] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=1152921504606847409, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a64060, count:5, allocator:0x7f92636887d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9264a64360, local_datums:0x7f9264a64360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:40.652942] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=56][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:40.653082] ERROR issue_dba_error (ob_log.cpp:1875) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.653108] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=25][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.653115] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:40.653118] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:40.653120] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 7, flag: 0, null: 0, ptr: 0x7f9264a68178, hex: 31303033343330, cstr: 1003430},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9264a6817f, hex: 3230323430393133303435303537, cstr: 20240913045057},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a680e0, hex: 6E48500300000000, int: 55593070},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68118, hex: 58B03BC5A1610BE8, int: -1726178684739801000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-26357, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:250, row_store_type:0, opt:5, var_column_count:0, row_offset:15394, original_length:16334, max_merged_trans_version:1726190643638412000, data_length:16334, data_zlength:5786, data_checksum:2846467531, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9287dce030, buf_size:5786, data_size:16334, row_count:250, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:796410710, row_count_delta:250, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16334}) [2024-09-26 16:00:40.653133] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=13][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:40.653135] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:40.653137] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:40.653138] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:40.653139] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9264a64360, local_datums:0x7f9264a64360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a643e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9264a64360, local_datums:0x7f9264a64360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a643e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9263688118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263688118, limit_param:null, stmt_allocator:0x7f92636887d8, allocator:0x7f92636887d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9264a64360, local_datums:0x7f9264a64360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a643e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337640626000, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:40.653218] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=78][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:40.653220] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9264a64360, local_datums:0x7f9264a64360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a643e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9263688118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263688118, limit_param:null, stmt_allocator:0x7f92636887d8, allocator:0x7f92636887d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9264a64360, local_datums:0x7f9264a64360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a643e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337640626000, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:40.653320] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=100][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:40.653323] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=2] partition merge iter row count(i=0, row_count=250, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337640626000, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.653335] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:40.653345] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:40.653348] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:40.653349] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337640652445, end_time=1727337640653349, runtime=904, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64B-0-0, dag_ret:0, dag_status:2, start_time:1727337640642629, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.653344] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337640625643, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.653356] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64B-0-0, dag_ret:0, dag_status:2, start_time:1727337640642629, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:40.653361] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=16] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:40.653364] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=10734, dag_cnt=37, dag_cnts_[dag.get_type()]=37, &dag=0x7f9264d78480, dag={ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640642629, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.653372] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640642625, task_id:YB42C0A8002D-0006230056D8C64B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.653365] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9262c77480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.653420] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337640626150, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.653437] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.653466] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=24] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.653469] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.653475] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:331}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.653500] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3214, total_rows=22498) [2024-09-26 16:00:40.653505] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.653494] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:331}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:331}, table_count:1, [{i:0, table_key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}, merge_scn:{val:1726241465393532000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8633ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640651601, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.87ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.653512] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9262c77480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847408}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.653521] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C651-0-0, dag_ret:0, dag_status:2, start_time:1727337640651570, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}, scanned_row_cnt_arr:0x7f9287db7480, output_block_cnt_arr:0x7f9287db7488, concurrent_cnt:1, estimate_row_cnt:45, estimate_occupy_size:0, latest_update_ts:1727337640651601, estimated_finish_time:1727337657653521}) [2024-09-26 16:00:40.653528] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7a760, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C651-0-0, dag_ret:0, dag_status:2, start_time:1727337640651570, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:40.653533] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=5] task finish process(ret=0, start_time=1727337640651599, end_time=1727337640653533, runtime=1934, *this={this:0x7f9264d7a760, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C651-0-0, dag_ret:0, dag_status:2, start_time:1727337640651570, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:40.653531] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:1152921504606847408}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9262c77480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847408}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640644123, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=9.39ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.653549] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=5] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.653556] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337640653554, task_id:YB42C0A8002D-0006230056D8C652-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.653560] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=3] schedule one task(task=0x7f9264d7ac90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.653580] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.653584] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.653586] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.653599] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=9] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.653603] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.653610] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.653631] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=331, range=0x7f92a9b52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d38060, count:5, allocator:0x7f92c18227d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9287d38360, local_datums:0x7f9287d38360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:40.653654] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64A-0-0, dag_ret:0, dag_status:2, start_time:1727337640642617, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9262c775d0, output_block_cnt_arr:0x7f9262c775d8, concurrent_cnt:1, estimate_row_cnt:35098, estimate_occupy_size:18874368, latest_update_ts:1727337640644123, estimated_finish_time:1727337657653662}) [2024-09-26 16:00:40.653663] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264d78310, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64A-0-0, dag_ret:0, dag_status:2, start_time:1727337640642617, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.653669] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=5] task finish process(ret=0, start_time=1727337640644121, end_time=1727337640653669, runtime=9548, *this={this:0x7f9264d78310, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64A-0-0, dag_ret:0, dag_status:2, start_time:1727337640642617, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.653844] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:40.653883] ERROR issue_dba_error (ob_log.cpp:1875) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.653886] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.653893] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:40.653895] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:40.653897] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68070, hex: 0000000000000000, int: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a680a8, hex: 3501000000000000, int: 309},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a680e0, hex: 3501000000000000, int: 309},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68118, hex: 7032CD318A310BE8, int: -1726231562555674000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-24587, column_count:14, rowkey_column_count:5, has_column_checksum:0, row_count:196, row_store_type:0, opt:0, var_column_count:0, row_offset:15642, original_length:16366, max_merged_trans_version:1726241465393532000, data_length:16366, data_zlength:16366, data_checksum:3184742917, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926f604090, buf_size:16366, data_size:16366, row_count:196, column_count:14, max_merged_trans_version:1726241465393532000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2306482841, row_count_delta:7, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16366}) [2024-09-26 16:00:40.653908] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=11][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:40.653910] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:40.653911] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:40.653913] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=2][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:40.653914] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:40.653916] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:40.653917] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9287d38360, local_datums:0x7f9287d38360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d384f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:331}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752994696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9287d38360, local_datums:0x7f9287d38360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d384f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, store_ctx:{this:0x7f92c1822118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c1822118, limit_param:null, stmt_allocator:0x7f92c18227d8, allocator:0x7f92c18227d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241465393532000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9287d38360, local_datums:0x7f9287d38360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d384f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, key:{"BIGINT":0},{"BIGINT":309},{"BIGINT":309}, value_iter:{value:{this=0x7f92830f5dd8 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=15 list_head=0x7f928848eeb8 latest_compact_node=0x7f9283113eb0 max_trans_version={val:1726241465393532000, v:0} max_trans_id=125582299 max_elr_trans_version={val:1726241465393532000, v:0} max_elr_trans_id=125582299 latest_compact_ts=1727334207881700 last_compact_cnt=49 total_trans_node_cnt=141 max_modify_scn={val:1726241465393532000, v:0} min_modify_scn={val:1726179340987317000, v:0}}, version_iter:this=0x7f928848eb78 trans_version={val:1726241465393532000, v:0} scn={val:1726241465393532000, v:0} tx_id={txid:125582299} prev=0x7f928848e850 next=0x7f928848eeb8 modify_count=65 acc_checksum=2418483105 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:42} seq_no=1726241465159151, multi_version_iter:this=0x7f9281a21200 trans_version={val:1726230662165479000, v:0} scn={val:1726230662165479000, v:0} tx_id={txid:125497658} prev=0x7f9281a20ec0 next=0x7f9283111860 modify_count=41 acc_checksum=883883390 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:76} seq_no=1726230662033825, max_committed_trans_version:1726241465393532000, cur_trans_version:{val:1726231562555674000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241465393532000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:196, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337640626150, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:14, cur_pos:0, count:0}}) [2024-09-26 16:00:40.653975] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=58][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:40.653976] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:331}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752994696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9287d38360, local_datums:0x7f9287d38360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d384f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, store_ctx:{this:0x7f92c1822118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c1822118, limit_param:null, stmt_allocator:0x7f92c18227d8, allocator:0x7f92c18227d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241465393532000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9287d38360, local_datums:0x7f9287d38360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d383e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d384f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d385d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, key:{"BIGINT":0},{"BIGINT":309},{"BIGINT":309}, value_iter:{value:{this=0x7f92830f5dd8 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=15 list_head=0x7f928848eeb8 latest_compact_node=0x7f9283113eb0 max_trans_version={val:1726241465393532000, v:0} max_trans_id=125582299 max_elr_trans_version={val:1726241465393532000, v:0} max_elr_trans_id=125582299 latest_compact_ts=1727334207881700 last_compact_cnt=49 total_trans_node_cnt=141 max_modify_scn={val:1726241465393532000, v:0} min_modify_scn={val:1726179340987317000, v:0}}, version_iter:this=0x7f928848eb78 trans_version={val:1726241465393532000, v:0} scn={val:1726241465393532000, v:0} tx_id={txid:125582299} prev=0x7f928848e850 next=0x7f928848eeb8 modify_count=65 acc_checksum=2418483105 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:42} seq_no=1726241465159151, multi_version_iter:this=0x7f9281a21200 trans_version={val:1726230662165479000, v:0} scn={val:1726230662165479000, v:0} tx_id={txid:125497658} prev=0x7f9281a20ec0 next=0x7f9283111860 modify_count=41 acc_checksum=883883390 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:76} seq_no=1726230662033825, max_committed_trans_version:1726241465393532000, cur_trans_version:{val:1726231562555674000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241465393532000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:196, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337640626150, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:14, cur_pos:0, count:0}}]) [2024-09-26 16:00:40.654025] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=49][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:40.654028] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=3] partition merge iter row count(i=0, row_count=196, ghost_row_count=0, pkey={tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, table={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337640626150, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.654040] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:40.654048] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:40.654051] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:40.654052] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337640653573, end_time=1727337640654052, runtime=479, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C651-0-0, dag_ret:0, dag_status:2, start_time:1727337640651570, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:40.654060] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C651-0-0, dag_ret:0, dag_status:2, start_time:1727337640651570, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:40.654067] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=2497, dag_cnt=36, dag_cnts_[dag.get_type()]=36, &dag=0x7f9264d7a3a0, dag={ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C651-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640651570, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}) [2024-09-26 16:00:40.654074] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C651-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640651560, task_id:YB42C0A8002D-0006230056D8C651-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.654103] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=2] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.654121] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] succeed to add sys task(task={start_time:1727337640654119, task_id:YB42C0A8002D-0006230056D8C653-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.654126] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=4] schedule one task(task=0x7f9264d7b1c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.654180] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=7] multi version scan iterator init succ(param.table_id_=1152921504606847408, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:5, allocator:0x7f92864067d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:40.654338] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337640626091, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.654352] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753327496, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753327496, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.654367] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.654371] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.654376] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:444}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753327496, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.654395] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:444}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753327496, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:444}, table_count:1, [{i:0, table_key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8632d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640650325, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.05ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.654419] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:40.654421] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64F-0-0, dag_ret:0, dag_status:2, start_time:1727337640650285, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3490, output_block_cnt_arr:0x7f9287db3498, concurrent_cnt:1, estimate_row_cnt:4, estimate_occupy_size:6291456, latest_update_ts:1727337640650325, estimated_finish_time:1727337657654424}) [2024-09-26 16:00:40.654446] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=25] succeed to init merge ctx(task={this:0x7f9264d79d00, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64F-0-0, dag_ret:0, dag_status:2, start_time:1727337640650285, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.654452] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=5] task finish process(ret=0, start_time=1727337640650323, end_time=1727337640654452, runtime=4129, *this={this:0x7f9264d79d00, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64F-0-0, dag_ret:0, dag_status:2, start_time:1727337640650285, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.654490] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.654494] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.654501] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:40.654503] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:40.654505] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9263b5a178, hex: 30323232362D32312D373825, cstr: 02226-21-78%},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9263b5a184, hex: 3230323430393133303435343039, cstr: 20240913045409},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a0e0, hex: CD7F500300000000, int: 55607245},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a118, hex: E8EC05CCA0600BE8, int: -1726179788432479000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-11869, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:238, row_store_type:0, opt:5, var_column_count:0, row_offset:15429, original_length:16321, max_merged_trans_version:1726190643638412000, data_length:16321, data_zlength:5388, data_checksum:152456761, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9264d58030, buf_size:5388, data_size:16321, row_count:238, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2114797869, row_count_delta:238, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16321}) [2024-09-26 16:00:40.654517] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:40.654518] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:40.654520] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:40.654521] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:40.654523] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a383e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a383e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9286406118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286406118, limit_param:null, stmt_allocator:0x7f92864067d8, allocator:0x7f92864067d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320384, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a383e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337640625643, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:40.654589] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=66][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:40.654591] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a383e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9286406118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286406118, limit_param:null, stmt_allocator:0x7f92864067d8, allocator:0x7f92864067d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320384, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a383e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337640625643, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:40.654626] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=34][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:40.654628] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=1] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337640625643, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.654627] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337640626075, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.654640] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:40.654641] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753247280, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828753247280, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.654649] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:40.654653] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:40.654652] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.654655] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.654654] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337640654139, end_time=1727337640654654, runtime=515, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64A-0-0, dag_ret:0, dag_status:2, start_time:1727337640642617, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.654661] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64A-0-0, dag_ret:0, dag_status:2, start_time:1727337640642617, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:40.654660] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:412}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753247280, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.654668] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=12051, dag_cnt=35, dag_cnts_[dag.get_type()]=35, &dag=0x7f9264d77f50, dag={ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640642617, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.654676] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64A-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337640642606, task_id:YB42C0A8002D-0006230056D8C64A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.654678] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:412}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753247280, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:412}, table_count:1, [{i:0, table_key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640649538, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.12ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.654699] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64E-0-0, dag_ret:0, dag_status:2, start_time:1727337640649436, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad470, output_block_cnt_arr:0x7f92684ad478, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337640649538, estimated_finish_time:1727337657654702}) [2024-09-26 16:00:40.654706] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d797d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64E-0-0, dag_ret:0, dag_status:2, start_time:1727337640649436, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.654709] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.654713] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.654711] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=5] task finish process(ret=0, start_time=1727337640649535, end_time=1727337640654711, runtime=5176, *this={this:0x7f9264d797d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64E-0-0, dag_ret:0, dag_status:2, start_time:1727337640649436, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.654715] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.654719] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.654722] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.654729] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.654742] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=2] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.654761] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=3] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.654767] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337640654765, task_id:YB42C0A8002D-0006230056D8C654-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.654770] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=3] schedule one task(task=0x7f9264d7b6f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.654787] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.654789] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.654791] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.654794] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.654796] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.654801] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.654832] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=412, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:7}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fa060, count:4, allocator:0x7f92bc9127d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f92665fa350, local_datums:0x7f92665fa350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:40.654833] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=444, range=0x7f92aae56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b58060, count:6, allocator:0x7f92636287d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9263b58390, local_datums:0x7f9263b58390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:40.654857] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:40.654907] ERROR issue_dba_error (ob_log.cpp:1875) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.654913] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.654920] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:40.654923] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=3][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.654927] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=0] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337640626075, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.654950] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640654800, end_time=1727337640654950, runtime=150, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64E-0-0, dag_ret:0, dag_status:2, start_time:1727337640649436, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.654960] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5524, dag_cnt=34, dag_cnts_[dag.get_type()]=34, &dag=0x7f9264d79410, dag={ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640649436, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.654968] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64E-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640649429, task_id:YB42C0A8002D-0006230056D8C64E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.655041] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.655077] ERROR issue_dba_error (ob_log.cpp:1875) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.655080] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.655090] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=0] partition merge iter row count(i=0, row_count=207, ghost_row_count=0, pkey={tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337640626091, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.655109] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640654774, end_time=1727337640655109, runtime=335, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64F-0-0, dag_ret:0, dag_status:2, start_time:1727337640650285, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.655119] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4834, dag_cnt=33, dag_cnts_[dag.get_type()]=33, &dag=0x7f9264d79940, dag={ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C64F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640650285, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.655160] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C64F-0-0] [lt=39] succeed to del sys task(removed_task={start_time:1727337640650278, task_id:YB42C0A8002D-0006230056D8C64F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.655317] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640655315, task_id:YB42C0A8002D-0006230056D8C655-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.655322] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=4] schedule one task(task=0x7f9264d7bc20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.655329] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640655328, task_id:YB42C0A8002D-0006230056D8C656-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.655332] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=2] schedule one task(task=0x7f9264d7c150, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.655351] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.655355] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.655357] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.655362] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.655365] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.655371] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.655403] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.655405] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.655407] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.655409] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.655411] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.655415] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.656125] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337640626132, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.656142] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752918752, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828752918752, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.656145] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337640626168, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.656157] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.656161] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.656155] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753357664, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753357664, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.656167] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:309}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752918752, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.656186] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:309}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752918752, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:309}, table_count:1, [{i:0, table_key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}, merge_scn:{val:1726241316899724000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8631da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640650427, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.74ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.656211] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C650-0-0, dag_ret:0, dag_status:2, start_time:1727337640650296, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5490, output_block_cnt_arr:0x7f9287db5498, concurrent_cnt:1, estimate_row_cnt:6, estimate_occupy_size:6291456, latest_update_ts:1727337640650427, estimated_finish_time:1727337657656214}) [2024-09-26 16:00:40.656215] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:40.656219] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.656218] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7a230, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C650-0-0, dag_ret:0, dag_status:2, start_time:1727337640650296, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:40.656223] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=5] task finish process(ret=0, start_time=1727337640650426, end_time=1727337640656223, runtime=5797, *this={this:0x7f9264d7a230, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C650-0-0, dag_ret:0, dag_status:2, start_time:1727337640650296, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:40.656224] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:458}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753357664, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.656243] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:458}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753357664, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:458}, table_count:1, [{i:0, table_key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8627b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640653574, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.65ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.656262] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=2] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.656325] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=309, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:11}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3c060, count:7, allocator:0x7f92865a27d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9263a3c3b0, local_datums:0x7f9263a3c3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:40.656376] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=21] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:40.656514] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.656551] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C652-0-0, dag_ret:0, dag_status:2, start_time:1727337640653560, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b4a0, output_block_cnt_arr:0x7f9263a4b4a8, concurrent_cnt:1, estimate_row_cnt:14598, estimate_occupy_size:4194304, latest_update_ts:1727337640653574, estimated_finish_time:1727337657656552}) [2024-09-26 16:00:40.656561] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264d7ac90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C652-0-0, dag_ret:0, dag_status:2, start_time:1727337640653560, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.656566] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.656566] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=5] task finish process(ret=0, start_time=1727337640653572, end_time=1727337640656566, runtime=2994, *this={this:0x7f9264d7ac90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C652-0-0, dag_ret:0, dag_status:2, start_time:1727337640653560, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.656570] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.656586] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=0] partition merge iter row count(i=0, row_count=156, ghost_row_count=0, pkey={tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, table={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337640626132, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.656609] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640656292, end_time=1727337640656609, runtime=317, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C650-0-0, dag_ret:0, dag_status:2, start_time:1727337640650296, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:40.656619] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=6323, dag_cnt=32, dag_cnts_[dag.get_type()]=32, &dag=0x7f9264d79e70, dag={ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C650-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640650296, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}) [2024-09-26 16:00:40.656626] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C650-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640650292, task_id:YB42C0A8002D-0006230056D8C650-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.656675] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=20] schedule one task(task=0x7f9272926030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.656682] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337640656681, task_id:YB42C0A8002D-0006230056D8C657-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.656685] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=3] schedule one task(task=0x7f9264d7c680, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.656729] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=458, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a64060, count:8, allocator:0x7f92691a27d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264a643e0, local_datums:0x7f9264a643e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:40.656800] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.656805] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.656808] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.656813] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.656816] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.656823] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.656959] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.656993] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.656996] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.657004] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=0] partition merge iter row count(i=0, row_count=247, ghost_row_count=0, pkey={tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337640626168, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.657024] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640656698, end_time=1727337640657023, runtime=325, *this={this:0x7f9272926030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C652-0-0, dag_ret:0, dag_status:2, start_time:1727337640653560, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.657032] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3472, dag_cnt=31, dag_cnts_[dag.get_type()]=31, &dag=0x7f9264d7a8d0, dag={ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C652-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640653560, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.657039] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C652-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640653554, task_id:YB42C0A8002D-0006230056D8C652-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.657088] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640657077, task_id:YB42C0A8002D-0006230056D8C658-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.657092] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=3] schedule one task(task=0x7f9264d7cbb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.657110] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.657113] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.657115] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726077706339704027) [2024-09-26 16:00:40.657119] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.657122] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.657129] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.657317] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337640626204, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.657335] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=17] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753332736, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=36, max_schema_version_in_memtable=1710828753332736, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.657545] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=12] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:40.657551] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.657558] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:455}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:10, rowkey_column_count:9, schema_rowkey_col_cnt:7, full_stored_col_cnt:38, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753332736, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=26 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.657591] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=27] succeed to build merge ctx(tablet_id={id:455}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753332736, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:455}, table_count:1, [{i:0, table_key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640654783, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.77ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.657752] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=21] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:40.657915] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C654-0-0, dag_ret:0, dag_status:2, start_time:1727337640654770, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d4b0, output_block_cnt_arr:0x7f9264b1d4b8, concurrent_cnt:1, estimate_row_cnt:3824, estimate_occupy_size:4194304, latest_update_ts:1727337640654783, estimated_finish_time:1727337657657916}) [2024-09-26 16:00:40.657927] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=11] succeed to init merge ctx(task={this:0x7f9264d7b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C654-0-0, dag_ret:0, dag_status:2, start_time:1727337640654770, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.657933] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=5] task finish process(ret=0, start_time=1727337640654781, end_time=1727337640657932, runtime=3151, *this={this:0x7f9264d7b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C654-0-0, dag_ret:0, dag_status:2, start_time:1727337640654770, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.657954] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.657976] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337640626186, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.657986] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753417144, storage_schema:{this:0x7f92684b3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828753417144, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.657992] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337640626257, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.658005] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753441792, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828753441792, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.658056] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:40.658058] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.658063] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1295, total_rows=1295) [2024-09-26 16:00:40.658066] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.658063] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684b3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:487}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753417144, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.658072] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:501}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753441792, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.658081] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:487}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753417144, storage_schema:{this:0x7f92684b3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:487}, table_count:1, [{i:0, table_key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640654704, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.36ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.658091] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:501}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753441792, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:501}, table_count:1, [{i:0, table_key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640655346, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.72ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.658166] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C656-0-0, dag_ret:0, dag_status:2, start_time:1727337640655331, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3490, output_block_cnt_arr:0x7f9287db3498, concurrent_cnt:1, estimate_row_cnt:1295, estimate_occupy_size:0, latest_update_ts:1727337640655346, estimated_finish_time:1727337657658166}) [2024-09-26 16:00:40.658174] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d7c150, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C656-0-0, dag_ret:0, dag_status:2, start_time:1727337640655331, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.658180] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=6] task finish process(ret=0, start_time=1727337640655344, end_time=1727337640658180, runtime=2836, *this={this:0x7f9264d7c150, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C656-0-0, dag_ret:0, dag_status:2, start_time:1727337640655331, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.658191] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=28] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:40.658201] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337640626220, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.658211] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753437040, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=57, max_schema_version_in_memtable=1710828753437040, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.658267] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1362, total_rows=1362) [2024-09-26 16:00:40.658269] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.658273] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:492}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:12, rowkey_column_count:11, schema_rowkey_col_cnt:9, full_stored_col_cnt:59, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753437040, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=28 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.658289] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C653-0-0, dag_ret:0, dag_status:2, start_time:1727337640654126, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684b34a0, output_block_cnt_arr:0x7f92684b34a8, concurrent_cnt:1, estimate_row_cnt:2199, estimate_occupy_size:0, latest_update_ts:1727337640654704, estimated_finish_time:1727337657658288}) [2024-09-26 16:00:40.658297] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C653-0-0, dag_ret:0, dag_status:2, start_time:1727337640654126, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.658302] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=4] task finish process(ret=0, start_time=1727337640654702, end_time=1727337640658302, runtime=3600, *this={this:0x7f9264d7b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C653-0-0, dag_ret:0, dag_status:2, start_time:1727337640654126, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.658297] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=20] succeed to build merge ctx(tablet_id={id:492}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753437040, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:492}, table_count:1, [{i:0, table_key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8624c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640655399, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.87ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.658333] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337640626313, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.658345] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753360568, storage_schema:{this:0x7f9287d37330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753360568, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.658353] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.658356] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.658360] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287d37330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:472}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753360568, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.658376] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:472}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753360568, storage_schema:{this:0x7f9287d37330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:472}, table_count:1, [{i:0, table_key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640656794, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.56ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.658387] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C655-0-0, dag_ret:0, dag_status:2, start_time:1727337640655321, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db74c0, output_block_cnt_arr:0x7f9287db74c8, concurrent_cnt:1, estimate_row_cnt:1362, estimate_occupy_size:2097152, latest_update_ts:1727337640655399, estimated_finish_time:1727337657658387}) [2024-09-26 16:00:40.658394] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7bc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C655-0-0, dag_ret:0, dag_status:2, start_time:1727337640655321, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.658395] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C657-0-0, dag_ret:0, dag_status:2, start_time:1727337640656685, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287d37460, output_block_cnt_arr:0x7f9287d37468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337640656794, estimated_finish_time:1727337657658395}) [2024-09-26 16:00:40.658399] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=5] task finish process(ret=0, start_time=1727337640655398, end_time=1727337640658399, runtime=3001, *this={this:0x7f9264d7bc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C655-0-0, dag_ret:0, dag_status:2, start_time:1727337640655321, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.658400] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7c680, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C657-0-0, dag_ret:0, dag_status:2, start_time:1727337640656685, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.658404] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=4] task finish process(ret=0, start_time=1727337640656791, end_time=1727337640658404, runtime=1613, *this={this:0x7f9264d7c680, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C657-0-0, dag_ret:0, dag_status:2, start_time:1727337640656685, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.658450] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=455, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:36, schema_rowkey_cnt:7, rowkey_cnt:9, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:7, column_cnt:38}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:9, allocator:0x7f92865547d8, capacity:9, init_cnt:9}, datum_utils:{is_oracle_mode:false, rowkey_cnt:9, col_cnt:9, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:38, datum_buffer:{capacity:38, datums:0x7f9263b5f340, local_datums:0x7f9263b5e3f0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,]}) [2024-09-26 16:00:40.658577] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:40.658720] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=39][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.658762] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.658765] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.658777] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=0] partition merge iter row count(i=0, row_count=140, ghost_row_count=0, pkey={tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337640626204, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.658799] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640658417, end_time=1727337640658799, runtime=382, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C654-0-0, dag_ret:0, dag_status:2, start_time:1727337640654770, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.658823] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4053, dag_cnt=30, dag_cnts_[dag.get_type()]=30, &dag=0x7f9264d7b330, dag={ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C654-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640654770, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.658832] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C654-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337640654765, task_id:YB42C0A8002D-0006230056D8C654-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.658878] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337640626330, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.658891] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753001832, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=21, max_schema_version_in_memtable=1710828753001832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.658902] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.658904] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.658909] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:342}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:23, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753001832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.658928] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:342}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753001832, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:342}, table_count:1, [{i:0, table_key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640657105, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.80ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.658948] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C658-0-0, dag_ret:0, dag_status:2, start_time:1727337640657092, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b490, output_block_cnt_arr:0x7f9263a4b498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337640657105, estimated_finish_time:1727337657658948}) [2024-09-26 16:00:40.658953] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C658-0-0, dag_ret:0, dag_status:2, start_time:1727337640657092, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.658958] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=4] task finish process(ret=0, start_time=1727337640657103, end_time=1727337640658958, runtime=1855, *this={this:0x7f9264d7cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C658-0-0, dag_ret:0, dag_status:2, start_time:1727337640657092, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.658970] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=24] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.658975] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=4] schedule one task(task=0x7f9264c803c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.658978] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=2] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.658981] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=1] schedule one task(task=0x7f9264af8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.658983] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=1] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.658987] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337640658986, task_id:YB42C0A8002D-0006230056D8C659-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.658991] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=3] schedule one task(task=0x7f9264d7d0e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.659024] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=472, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:3, allocator:0x7f92bc8f67d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f92665f8320, local_datums:0x7f92665f8320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:40.659030] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=487, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:11}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:8, allocator:0x7f92bc8f87d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9264a623e0, local_datums:0x7f9264a623e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:40.659062] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.659101] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.659104] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.659113] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=0] partition merge iter row count(i=0, row_count=17, ghost_row_count=0, pkey={tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337640626313, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.659133] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640658990, end_time=1727337640659133, runtime=143, *this={this:0x7f9264af8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C657-0-0, dag_ret:0, dag_status:2, start_time:1727337640656685, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.659142] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2456, dag_cnt=29, dag_cnts_[dag.get_type()]=29, &dag=0x7f9264d7c2c0, dag={ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C657-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640656685, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.659149] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C657-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640656681, task_id:YB42C0A8002D-0006230056D8C657-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.659265] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.659300] ERROR issue_dba_error (ob_log.cpp:1875) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.659303] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.659313] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=0] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337640626186, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.659334] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640659000, end_time=1727337640659334, runtime=334, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C653-0-0, dag_ret:0, dag_status:2, start_time:1727337640654126, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.659346] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5220, dag_cnt=28, dag_cnts_[dag.get_type()]=28, &dag=0x7f9264d7ae00, dag={ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C653-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640654126, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.659354] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C653-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337640654119, task_id:YB42C0A8002D-0006230056D8C653-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.659361] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.659367] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.659370] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.659379] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337640659377, task_id:YB42C0A8002D-0006230056D8C65A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.659382] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=2] schedule one task(task=0x7f9264d7d610, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.659394] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.659403] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640659401, task_id:YB42C0A8002D-0006230056D8C65B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.659406] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=2] schedule one task(task=0x7f9264d7db40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.659415] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.659419] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.659421] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.659425] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.659429] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.659436] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.659399] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8622d40, is_inited:true, wash_score:1727337640625574000, hold_ref_cnt:true, ref_cnt:2, version:2, length:492, tablet_addr:[816017-1849-0-2305843631985766400-2](offset=1814528,size=580,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:140}, data_tablet_id:{id:140}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726073917016589000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2305242685, row_count:150}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86233d4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844315959259136-0](offset=1765376,size=1217,type=2,seq=0), type:"MAJOR", tablet_id:{id:140}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726070403667750001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843943370847425-0](offset=1766593,size=870,type=2,seq=0), type:"MINOR", tablet_id:{id:140}, scn_range:{start_scn:{val:1726073917016589000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726167395874925000, upper_trans_version:1726167395874925000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337640626349, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcfc00, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495712}, this:0x7f92a3bcfc00, timestamp:1727334209495712, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:57, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726256203821911000, v:0}, rec_scn:{val:1726241803398655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697781760-2](offset=1802240,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834319180-0](offset=1802572,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881148892-0](offset=1802716,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447668926-0](offset=1802942,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7b430, 0x7f92a3bcfc00, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.659485] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.659494] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.659546] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=342, range=0x7f92a9ed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:21, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:23}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a60060, count:6, allocator:0x7f92a31fc7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:23, datum_buffer:{capacity:32, datums:0x7f9264a60390, local_datums:0x7f9264a60390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,]}) [2024-09-26 16:00:40.659578] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.659617] ERROR issue_dba_error (ob_log.cpp:1875) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.659619] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.659628] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337640626330, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.659649] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640659512, end_time=1727337640659648, runtime=136, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C658-0-0, dag_ret:0, dag_status:2, start_time:1727337640657092, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.659659] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2567, dag_cnt=27, dag_cnts_[dag.get_type()]=27, &dag=0x7f9264d7c7f0, dag={ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C658-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640657092, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.659667] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C658-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337640657077, task_id:YB42C0A8002D-0006230056D8C658-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.659719] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=501, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:6, allocator:0x7f92b33d87d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263a38390, local_datums:0x7f9263a38390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:40.659749] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337640659745, task_id:YB42C0A8002D-0006230056D8C65C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.659757] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=7] schedule one task(task=0x7f9264d7e070, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.659780] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.659785] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.659787] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:40.659791] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.659792] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.659796] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.659803] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.659834] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.659837] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.659846] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=0] partition merge iter row count(i=0, row_count=56, ghost_row_count=0, pkey={tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337640626257, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.659865] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640659694, end_time=1727337640659865, runtime=171, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C656-0-0, dag_ret:0, dag_status:2, start_time:1727337640655331, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.659874] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4543, dag_cnt=26, dag_cnts_[dag.get_type()]=26, &dag=0x7f9264d7bd90, dag={ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C656-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640655331, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.659881] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C656-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640655328, task_id:YB42C0A8002D-0006230056D8C656-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.659953] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=492, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:57, schema_rowkey_cnt:9, rowkey_cnt:11, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:9, column_cnt:59}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5c060, count:11, allocator:0x7f92a31fa7d8, capacity:11, init_cnt:11}, datum_utils:{is_oracle_mode:false, rowkey_cnt:11, col_cnt:11, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:59, datum_buffer:{capacity:59, datums:0x7f9263a3e050, local_datums:0x7f9263b5c440}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,col_id=43:NOP,col_id=44:NOP,col_id=45:NOP,col_id=46:NOP,col_id=47:NOP,col_id=48:NOP,col_id=49:NOP,col_id=50:NOP,col_id=51:NOP,col_id=52:NOP,col_id=53:NOP,col_id=54:NOP,col_id=55:NOP,col_id=56:NOP,col_id=57:NOP,col_id=58:NOP,]}) [2024-09-26 16:00:40.659967] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337640659963, task_id:YB42C0A8002D-0006230056D8C65D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.659975] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=7] schedule one task(task=0x7f9264d7e5a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.660002] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.660007] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.660009] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:40.660014] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.660017] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.660024] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.660148] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.660198] ERROR issue_dba_error (ob_log.cpp:1875) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.660201] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.660209] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=0] partition merge iter row count(i=0, row_count=61, ghost_row_count=0, pkey={tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337640626220, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.660226] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640659925, end_time=1727337640660226, runtime=301, *this={this:0x7f9264c803c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C655-0-0, dag_ret:0, dag_status:2, start_time:1727337640655321, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.660234] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4913, dag_cnt=25, dag_cnts_[dag.get_type()]=25, &dag=0x7f9264d7b860, dag={ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C655-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640655321, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.660241] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C655-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640655315, task_id:YB42C0A8002D-0006230056D8C655-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.660271] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.660274] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.660276] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:40.660280] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.660309] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337640660305, task_id:YB42C0A8002D-0006230056D8C65E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.660317] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=7] schedule one task(task=0x7f9264d7ead0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.660283] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8642d40, is_inited:true, wash_score:1727337640625574000, hold_ref_cnt:true, ref_cnt:2, version:2, length:487, tablet_addr:[816017-1849-0-2305843626617085952-2](offset=1843200,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101010}, data_tablet_id:{id:140}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726073917016589000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4294280206, row_count:150}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86433f8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844211806371840-0](offset=1835008,size=1120,type=2,seq=0), type:"MAJOR", tablet_id:{id:101010}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726070403667750001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843968066978912-0](offset=1836128,size=893,type=2,seq=0), type:"MINOR", tablet_id:{id:101010}, scn_range:{start_scn:{val:1726073917016589000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726167395874925000, upper_trans_version:1726167395874925000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337640626403, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd1370, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495895}, this:0x7f92a3bd1370, timestamp:1727334209495895, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:60, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726256203821911000, v:0}, rec_scn:{val:1726241803398655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697818624-2](offset=1839104,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276806988-0](offset=1839436,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881185762-0](offset=1839586,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7c3d0, 0x7f92a3bd1370, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.660323] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=40] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.660329] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.660345] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.660349] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.660351] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:40.660356] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.660360] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864eec0, is_inited:true, wash_score:1727337640625574000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[814042-5586-0-2305843650238771200-2](offset=1208320,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:297}, data_tablet_id:{id:297}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726063409221604000, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2361333062, row_count:24}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844531780800512-0](offset=1200128,size=1418,type=2,seq=0), type:"MAJOR", tablet_id:{id:297}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063202518238000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843919747962250-0](offset=1201546,size=848,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726063409221604000, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557514739000, upper_trans_version:1726121557514739000}, {i:1, addr:[814042-5586-0-2305843920821704922-0](offset=1202394,size=849,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149603136610000, upper_trans_version:1726149603136610000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337640626453, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd86d0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220478269}, this:0x7f92a3bd86d0, timestamp:1727334220478269, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:74, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726264804632655000, v:0}, rec_scn:{val:1726264804632655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513817088-2](offset=1204224,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843183161074102-0](offset=1204662,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880551000-0](offset=1204824,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447071034-0](offset=1205050,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7eae0, 0x7f92a3bd86d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.660407] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=47] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.660414] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.660567] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337640626387, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.660580] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.660601] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.660605] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.660611] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101009}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:40.660638] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=20] succeed to build merge ctx(tablet_id={id:101009}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101009}, table_count:1, [{i:0, table_key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8641da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640659410, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.20ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.660618] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337640626349, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.660675] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=29] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65A-0-0, dag_ret:0, dag_status:2, start_time:1727337640659382, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1470, output_block_cnt_arr:0x7f92728f1478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337640659410, estimated_finish_time:1727337657660675}) [2024-09-26 16:00:40.660674] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=55] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.660682] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7d610, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65A-0-0, dag_ret:0, dag_status:2, start_time:1727337640659382, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.660687] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.660688] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=5] task finish process(ret=0, start_time=1727337640659408, end_time=1727337640660687, runtime=1279, *this={this:0x7f9264d7d610, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65A-0-0, dag_ret:0, dag_status:2, start_time:1727337640659382, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.660691] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.660698] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=4] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.660696] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:140}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.660717] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337640626435, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.660728] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=41, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.660721] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=20] succeed to build merge ctx(tablet_id={id:140}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:140}, table_count:1, [{i:0, table_key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8622d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640659354, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.34ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.660741] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.660745] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.660748] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C659-0-0, dag_ret:0, dag_status:2, start_time:1727337640658990, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d460, output_block_cnt_arr:0x7f9264b1d468, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337640659354, estimated_finish_time:1727337657660747}) [2024-09-26 16:00:40.660754] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C659-0-0, dag_ret:0, dag_status:2, start_time:1727337640658990, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.660759] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=5] task finish process(ret=0, start_time=1727337640659351, end_time=1727337640660759, runtime=1408, *this={this:0x7f9264d7d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C659-0-0, dag_ret:0, dag_status:2, start_time:1727337640658990, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.660750] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:377}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:43, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.660754] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101009, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:4, allocator:0x7f92865b67d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665f8350, local_datums:0x7f92665f8350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:40.660789] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.660780] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:377}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:377}, table_count:1, [{i:0, table_key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640659995, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=753us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.660779] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337640626403, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.660811] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=32] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.660805] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65D-0-0, dag_ret:0, dag_status:2, start_time:1727337640659974, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad480, output_block_cnt_arr:0x7f92684ad488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337640659995, estimated_finish_time:1727337657660804}) [2024-09-26 16:00:40.660821] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.660820] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=15] succeed to init merge ctx(task={this:0x7f9264d7e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65D-0-0, dag_ret:0, dag_status:2, start_time:1727337640659974, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.660824] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.660827] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.660826] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=5] task finish process(ret=0, start_time=1727337640659992, end_time=1727337640660825, runtime=833, *this={this:0x7f9264d7e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65D-0-0, dag_ret:0, dag_status:2, start_time:1727337640659974, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.660829] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.660828] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101010}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:40.660839] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337640626387, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.660843] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337640626453, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.660843] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:101010}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101010}, table_count:1, [{i:0, table_key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8642d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640660266, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=560us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.660852] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=16, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.660860] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.660863] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.660861] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65B-0-0, dag_ret:0, dag_status:2, start_time:1727337640659405, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685e7470, output_block_cnt_arr:0x7f92685e7478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337640660266, estimated_finish_time:1727337657660861}) [2024-09-26 16:00:40.660860] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640660721, end_time=1727337640660859, runtime=138, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65A-0-0, dag_ret:0, dag_status:2, start_time:1727337640659382, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.660867] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7db40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65B-0-0, dag_ret:0, dag_status:2, start_time:1727337640659405, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.660870] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1488, dag_cnt=24, dag_cnts_[dag.get_type()]=24, &dag=0x7f9264d7d250, dag={ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640659382, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.660872] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=4] task finish process(ret=0, start_time=1727337640660264, end_time=1727337640660871, runtime=607, *this={this:0x7f9264d7db40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65B-0-0, dag_ret:0, dag_status:2, start_time:1727337640659405, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.660867] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:297}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:18, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.660878] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65A-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640659377, task_id:YB42C0A8002D-0006230056D8C65A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.660883] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:297}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:297}, table_count:1, [{i:0, table_key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640660339, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=526us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.660899] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65E-0-0, dag_ret:0, dag_status:2, start_time:1727337640660316, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b490, output_block_cnt_arr:0x7f9263a4b498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337640660339, estimated_finish_time:1727337657660898}) [2024-09-26 16:00:40.660903] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=4] succeed to init merge ctx(task={this:0x7f9264d7ead0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65E-0-0, dag_ret:0, dag_status:2, start_time:1727337640660316, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.660908] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=4] task finish process(ret=0, start_time=1727337640660336, end_time=1727337640660907, runtime=571, *this={this:0x7f9264d7ead0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65E-0-0, dag_ret:0, dag_status:2, start_time:1727337640660316, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.660906] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337640626418, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.660918] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=4] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.660914] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752949376, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=20, max_schema_version_in_memtable=1710828752949376, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.660924] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=3] schedule one task(task=0x7f9264d852e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.660929] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=2] schedule one task(task=0x7f9264d85670, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.660933] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640660931, task_id:YB42C0A8002D-0006230056D8C65F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.660937] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=3] schedule one task(task=0x7f9264d7f000, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.660945] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.660948] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.660954] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:323}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:22, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752949376, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.660972] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=140, range=0x7f92aa6d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c72060, count:3, allocator:0x7f92bc9127d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9262c72320, local_datums:0x7f9262c72320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:40.660987] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=27] succeed to build merge ctx(tablet_id={id:323}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752949376, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:323}, table_count:1, [{i:0, table_key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640659774, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.18ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.661028] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.661034] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65C-0-0, dag_ret:0, dag_status:2, start_time:1727337640659756, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab480, output_block_cnt_arr:0x7f92684ab488, concurrent_cnt:1, estimate_row_cnt:205, estimate_occupy_size:0, latest_update_ts:1727337640659774, estimated_finish_time:1727337657661033}) [2024-09-26 16:00:40.661037] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.661041] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.661043] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:40.661041] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7e070, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65C-0-0, dag_ret:0, dag_status:2, start_time:1727337640659756, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.661047] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.661046] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=5] task finish process(ret=0, start_time=1727337640659772, end_time=1727337640661046, runtime=1274, *this={this:0x7f9264d7e070, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65C-0-0, dag_ret:0, dag_status:2, start_time:1727337640659756, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.661051] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.661055] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=1] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.661057] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.661065] ERROR issue_dba_error (ob_log.cpp:1875) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.661068] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.661078] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337640626349, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.661112] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640660940, end_time=1727337640661112, runtime=172, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C659-0-0, dag_ret:0, dag_status:2, start_time:1727337640658990, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.661109] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=297, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:16, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:18}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:6, allocator:0x7f92636267d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:18, datum_buffer:{capacity:32, datums:0x7f9263a3e390, local_datums:0x7f9263a3e390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,]}) [2024-09-26 16:00:40.661129] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2138, dag_cnt=23, dag_cnts_[dag.get_type()]=23, &dag=0x7f9264d7cd20, dag={ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C659-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640658990, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.661136] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C659-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640658986, task_id:YB42C0A8002D-0006230056D8C659-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.661138] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.661133] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=323, range=0x7f92aaf52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:20, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:22}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c76060, count:5, allocator:0x7f92ba7f67d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f9262c76360, local_datums:0x7f9262c76360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,]}) [2024-09-26 16:00:40.661171] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.661174] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.661182] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337640626453, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.661199] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640661069, end_time=1727337640661199, runtime=130, *this={this:0x7f9264d85670, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65E-0-0, dag_ret:0, dag_status:2, start_time:1727337640660316, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.661208] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=891, dag_cnt=22, dag_cnts_[dag.get_type()]=22, &dag=0x7f9264d7e710, dag={ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640660316, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.661215] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65E-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640660305, task_id:YB42C0A8002D-0006230056D8C65E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.661271] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=377, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:41, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:43}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a60060, count:5, allocator:0x7f92636267d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:43, datum_buffer:{capacity:43, datums:0x7f9264a612b0, local_datums:0x7f9264a60360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,]}) [2024-09-26 16:00:40.661307] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.661335] ERROR issue_dba_error (ob_log.cpp:1875) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.661338] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.661346] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=0] partition merge iter row count(i=0, row_count=6, ghost_row_count=0, pkey={tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337640626435, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.661353] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.661361] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640661244, end_time=1727337640661361, runtime=117, *this={this:0x7f9264d852e0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65D-0-0, dag_ret:0, dag_status:2, start_time:1727337640659974, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.661369] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1395, dag_cnt=21, dag_cnts_[dag.get_type()]=21, &dag=0x7f9264d7e1e0, dag={ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640659974, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.661375] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65D-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640659963, task_id:YB42C0A8002D-0006230056D8C65D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.661393] ERROR issue_dba_error (ob_log.cpp:1875) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.661396] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.661413] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=1] partition merge iter row count(i=0, row_count=171, ghost_row_count=0, pkey={tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337640626418, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.661445] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640661067, end_time=1727337640661445, runtime=378, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65C-0-0, dag_ret:0, dag_status:2, start_time:1727337640659756, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.661455] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1699, dag_cnt=20, dag_cnts_[dag.get_type()]=20, &dag=0x7f9264d7dcb0, dag={ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640659756, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.661464] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65C-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640659745, task_id:YB42C0A8002D-0006230056D8C65C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.661503] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=2] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.661510] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337640661509, task_id:YB42C0A8002D-0006230056D8C660-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.661513] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=3] schedule one task(task=0x7f9264d7f530, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.661517] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640661516, task_id:YB42C0A8002D-0006230056D8C661-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.661519] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=2] schedule one task(task=0x7f9264d7fa60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.661522] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337640661522, task_id:YB42C0A8002D-0006230056D8C662-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.661524] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=2] schedule one task(task=0x7f9264d7ff90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.661527] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640661527, task_id:YB42C0A8002D-0006230056D8C663-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.661529] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=2] schedule one task(task=0x7f9264d804c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.661531] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.661535] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.661538] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:40.661542] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.661545] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.661546] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.661548] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.661549] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.661550] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:40.661551] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:40.661555] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.661555] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.661559] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.661558] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.661565] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.661566] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.661583] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.661546] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706139416000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8651da0, is_inited:true, wash_score:1727337640625574000, hold_ref_cnt:true, ref_cnt:2, version:2, length:516, tablet_addr:[809012-1445-0-2305843657754255360-2](offset=499712,size=604,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:333}, data_tablet_id:{id:333}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706139416000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2218031597, row_count:2118}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86524c4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844845312692224-0](offset=479232,size=1710,type=2,seq=0), type:"MAJOR", tablet_id:{id:333}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843972360591022-0](offset=480942,size=897,type=2,seq=0), type:"MINI", tablet_id:{id:333}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337640626529, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdcd20, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220937046}, this:0x7f92a3bdcd20, timestamp:1727334220937046, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:83, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265274283037001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843367843950592-2](offset=487424,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843189602808142-0](offset=487758,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879834102-0](offset=487926,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446354136-0](offset=488152,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc4850, 0x7f92a3bdcd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.661585] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.661587] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:40.661585] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=39] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.661589] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.661592] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.661592] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077691154184001, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8650e00, is_inited:true, wash_score:1727337640625574000, hold_ref_cnt:true, ref_cnt:2, version:2, length:497, tablet_addr:[809012-1445-0-2305843637353218048-2](offset=557056,size=585,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:451}, data_tablet_id:{id:451}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077691154184001, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2119849913, row_count:31}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86514b8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844404004851712-0](offset=528384,size=1299,type=2,seq=0), type:"MAJOR", tablet_id:{id:451}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563069147000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843980950574355-0](offset=529683,size=905,type=2,seq=0), type:"MINI", tablet_id:{id:451}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986382094000, upper_trans_version:1726149986382094000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337640626586, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdfc00, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334222657108}, this:0x7f92a3bdfc00, timestamp:1727334222657108, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753331112, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:89, clock:641728512}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293246986000, v:0}, rec_scn:{val:1726265293246986000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696516096-2](offset=536576,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843170275504460-0](offset=536908,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879883234-0](offset=537058,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446403268-0](offset=537284,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc4080, 0x7f92a3bdfc00, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.661630] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=38] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.661635] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.662455] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:40.662478] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.663170] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101010, range=0x7f92ab056200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:4, allocator:0x7f92636267d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92664f0350, local_datums:0x7f92664f0350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:40.663215] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=18][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.663254] ERROR issue_dba_error (ob_log.cpp:1875) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.663257] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.663267] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337640626403, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.663339] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640663135, end_time=1727337640663339, runtime=204, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65B-0-0, dag_ret:0, dag_status:2, start_time:1727337640659405, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.663353] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3947, dag_cnt=19, dag_cnts_[dag.get_type()]=19, &dag=0x7f9264d7d780, dag={ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640659405, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.663374] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65B-0-0] [lt=17] succeed to del sys task(removed_task={start_time:1727337640659401, task_id:YB42C0A8002D-0006230056D8C65B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.663447] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640663444, task_id:YB42C0A8002D-0006230056D8C664-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.663454] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=6] schedule one task(task=0x7f9264d809f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.663491] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.663495] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.663497] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:40.663501] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.663504] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.663510] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.663550] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337640626488, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.663572] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=21] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92665fd330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.663587] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.663592] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.663598] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92665fd330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101065}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.663617] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101065}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92665fd330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101065}, table_count:1, [{i:0, table_key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640661032, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.56ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.663680] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=55] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65F-0-0, dag_ret:0, dag_status:2, start_time:1727337640660936, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665fd490, output_block_cnt_arr:0x7f92665fd498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337640661032, estimated_finish_time:1727337657663679}) [2024-09-26 16:00:40.663687] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7f000, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65F-0-0, dag_ret:0, dag_status:2, start_time:1727337640660936, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.663686] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337640626586, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.663692] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=5] task finish process(ret=0, start_time=1727337640661030, end_time=1727337640663692, runtime=2662, *this={this:0x7f9264d7f000, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65F-0-0, dag_ret:0, dag_status:2, start_time:1727337640660936, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.663697] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f92665ff330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=14, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.663704] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=4] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.663710] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.663724] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.663730] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92665ff330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:451}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:16, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.663747] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:451}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f92665ff330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:451}, table_count:1, [{i:0, table_key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8650e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640661580, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.15ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.663770] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C663-0-0, dag_ret:0, dag_status:2, start_time:1727337640661529, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665ff470, output_block_cnt_arr:0x7f92665ff478, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337640661580, estimated_finish_time:1727337657663769}) [2024-09-26 16:00:40.663775] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d804c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C663-0-0, dag_ret:0, dag_status:2, start_time:1727337640661529, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.663780] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=5] task finish process(ret=0, start_time=1727337640661579, end_time=1727337640663780, runtime=2201, *this={this:0x7f9264d804c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C663-0-0, dag_ret:0, dag_status:2, start_time:1727337640661529, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.663789] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=7] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.663809] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337640626529, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.663826] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101065, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a60060, count:6, allocator:0x7f92636267d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9264a60390, local_datums:0x7f9264a60390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:40.663833] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=23] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.663854] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.663857] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.663850] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=451, range=0x7f92ab152200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:14, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:16}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b04060, count:4, allocator:0x7f92b33fe7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:16, datum_buffer:{capacity:32, datums:0x7f9265b04350, local_datums:0x7f9265b04350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,]}) [2024-09-26 16:00:40.663868] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.663865] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:333}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.663879] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.663886] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:333}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:333}, table_count:1, [{i:0, table_key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8651da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640661526, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.34ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.663906] ERROR issue_dba_error (ob_log.cpp:1875) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.663908] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.663916] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C660-0-0, dag_ret:0, dag_status:2, start_time:1727337640661513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3490, output_block_cnt_arr:0x7f9287db3498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337640661526, estimated_finish_time:1727337657663916}) [2024-09-26 16:00:40.663923] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7f530, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C660-0-0, dag_ret:0, dag_status:2, start_time:1727337640661513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.663917] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=0] partition merge iter row count(i=0, row_count=15, ghost_row_count=0, pkey={tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337640626488, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.663928] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=4] task finish process(ret=0, start_time=1727337640661524, end_time=1727337640663928, runtime=2404, *this={this:0x7f9264d7f530, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C660-0-0, dag_ret:0, dag_status:2, start_time:1727337640661513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.663932] ERROR issue_dba_error (ob_log.cpp:1875) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.663936] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640663798, end_time=1727337640663936, runtime=138, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65F-0-0, dag_ret:0, dag_status:2, start_time:1727337640660936, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.663945] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3008, dag_cnt=18, dag_cnts_[dag.get_type()]=18, &dag=0x7f9264d7ec40, dag={ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C65F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640660936, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.663952] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C65F-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640660931, task_id:YB42C0A8002D-0006230056D8C65F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.663943] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=10][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.663961] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337640626586, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.663979] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337640626566, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.663988] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640663805, end_time=1727337640663988, runtime=183, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C663-0-0, dag_ret:0, dag_status:2, start_time:1727337640661529, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.664001] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=22] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=19, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.664013] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2483, dag_cnt=17, dag_cnts_[dag.get_type()]=17, &dag=0x7f9264d80100, dag={ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C663-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640661529, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.664021] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C663-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640661527, task_id:YB42C0A8002D-0006230056D8C663-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.664039] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.664042] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.664057] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337640626549, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.664056] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=11] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:334}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:21, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.664066] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.664080] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.664082] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.664076] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:334}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:334}, table_count:1, [{i:0, table_key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8654c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640661539, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.52ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.664087] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101066}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.664105] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101066}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101066}, table_count:1, [{i:0, table_key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8653ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640661541, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.54ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.664128] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C662-0-0, dag_ret:0, dag_status:2, start_time:1727337640661524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a434a0, output_block_cnt_arr:0x7f9263a434a8, concurrent_cnt:1, estimate_row_cnt:993, estimate_occupy_size:0, latest_update_ts:1727337640661539, estimated_finish_time:1727337657664128}) [2024-09-26 16:00:40.664133] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7ff90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C662-0-0, dag_ret:0, dag_status:2, start_time:1727337640661524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.664137] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C661-0-0, dag_ret:0, dag_status:2, start_time:1727337640661519, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665a7490, output_block_cnt_arr:0x7f92665a7498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337640661541, estimated_finish_time:1727337657664136}) [2024-09-26 16:00:40.664138] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=5] task finish process(ret=0, start_time=1727337640661538, end_time=1727337640664138, runtime=2600, *this={this:0x7f9264d7ff90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C662-0-0, dag_ret:0, dag_status:2, start_time:1727337640661524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.664143] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7fa60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C661-0-0, dag_ret:0, dag_status:2, start_time:1727337640661519, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.664148] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=4] task finish process(ret=0, start_time=1727337640661539, end_time=1727337640664148, runtime=2609, *this={this:0x7f9264d7fa60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C661-0-0, dag_ret:0, dag_status:2, start_time:1727337640661519, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.664155] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337640626623, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.664162] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=3] schedule one task(task=0x7f92665d63c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.664163] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.664168] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=3] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.664171] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.664172] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=3] schedule one task(task=0x7f9272926030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.664176] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640664175, task_id:YB42C0A8002D-0006230056D8C665-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.664180] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=3] schedule one task(task=0x7f9264d80f20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.664185] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337640664184, task_id:YB42C0A8002D-0006230056D8C666-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.664187] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=2] schedule one task(task=0x7f9264d81450, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.664192] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.664196] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:378}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.664210] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:378}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:378}, table_count:1, [{i:0, table_key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640663486, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=709us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.664226] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C664-0-0, dag_ret:0, dag_status:2, start_time:1727337640663453, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b480, output_block_cnt_arr:0x7f9263a4b488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337640663486, estimated_finish_time:1727337657664226}) [2024-09-26 16:00:40.664219] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=333, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:7, allocator:0x7f92636267d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f92684b43b0, local_datums:0x7f92684b43b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:40.664222] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101066, range=0x7f92a9b52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:7}, cols_desc:{ObIArray:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dce060, count:7, allocator:0x7f92b32387d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9287dce3b0, local_datums:0x7f9287dce3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:40.664231] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d809f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C664-0-0, dag_ret:0, dag_status:2, start_time:1727337640663453, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.664235] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=4] task finish process(ret=0, start_time=1727337640663484, end_time=1727337640664235, runtime=751, *this={this:0x7f9264d809f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C664-0-0, dag_ret:0, dag_status:2, start_time:1727337640663453, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.664369] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.664379] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.664403] ERROR issue_dba_error (ob_log.cpp:1875) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.664405] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.664418] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.664414] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=0] partition merge iter row count(i=0, row_count=182, ghost_row_count=0, pkey={tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337640626549, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.664421] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.664432] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640664198, end_time=1727337640664432, runtime=234, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C661-0-0, dag_ret:0, dag_status:2, start_time:1727337640661519, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.664431] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=0] partition merge iter row count(i=0, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337640626529, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.664441] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2922, dag_cnt=16, dag_cnts_[dag.get_type()]=16, &dag=0x7f9264d7f6a0, dag={ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C661-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640661519, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.664450] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C661-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337640661516, task_id:YB42C0A8002D-0006230056D8C661-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.664452] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640664182, end_time=1727337640664452, runtime=270, *this={this:0x7f92665d63c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C660-0-0, dag_ret:0, dag_status:2, start_time:1727337640661513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.664462] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2949, dag_cnt=15, dag_cnts_[dag.get_type()]=15, &dag=0x7f9264d7f170, dag={ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C660-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640661513, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.664469] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C660-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640661509, task_id:YB42C0A8002D-0006230056D8C660-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.664501] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.664504] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.664506] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:40.664499] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=334, range=0x7f92aaed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:19, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:21}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264b22060, count:8, allocator:0x7f92bc8f87d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:21, datum_buffer:{capacity:32, datums:0x7f9264b223e0, local_datums:0x7f9264b223e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,]}) [2024-09-26 16:00:40.664510] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.664515] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.664522] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241548806527000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.664731] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.664763] ERROR issue_dba_error (ob_log.cpp:1875) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.664766] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.664775] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=0] partition merge iter row count(i=0, row_count=153, ghost_row_count=0, pkey={tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337640626566, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.664793] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640664471, end_time=1727337640664793, runtime=322, *this={this:0x7f9272926030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C662-0-0, dag_ret:0, dag_status:2, start_time:1727337640661524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.664801] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3277, dag_cnt=14, dag_cnts_[dag.get_type()]=14, &dag=0x7f9264d7fbd0, dag={ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C662-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640661524, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.664807] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C662-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640661522, task_id:YB42C0A8002D-0006230056D8C662-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.664835] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.664838] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.664840] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:40.664844] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.664847] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b861fe60, is_inited:true, wash_score:1727337640626660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:491, tablet_addr:[816017-1849-0-2305843630912028672-2](offset=1818624,size=579,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:215}, data_tablet_id:{id:215}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3095927781, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86204f4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844118390796288-0](offset=1798144,size=1033,type=2,seq=0), type:"MAJOR", tablet_id:{id:215}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389038459000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843910084883465-0](offset=1799177,size=839,type=2,seq=0), type:"MINOR", tablet_id:{id:215}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726209400113641000, upper_trans_version:1726209400113641000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337640626669, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcf430, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209475373}, this:0x7f92a3bcf430, timestamp:1727334209475373, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:56, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277800266460000, v:0}, rec_scn:{val:1726241800098150000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697789952-2](offset=1810432,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834327372-0](offset=1810764,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881157084-0](offset=1810908,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447677118-0](offset=1811134,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc86d0, 0x7f92a3bcf430, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.664888] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=40] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.664894] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.664912] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=2] schedule one task(task=0x7f92729263c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.664917] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337640664916, task_id:YB42C0A8002D-0006230056D8C667-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.664920] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=3] schedule one task(task=0x7f9264d81980, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.664926] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640664925, task_id:YB42C0A8002D-0006230056D8C668-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.664928] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=2] schedule one task(task=0x7f9264d81eb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.664931] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640664931, task_id:YB42C0A8002D-0006230056D8C669-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.664933] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=2] schedule one task(task=0x7f9264d82460, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.664947] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.664951] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.664953] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:40.664957] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.664960] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.664966] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.664969] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=378, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b2060, count:5, allocator:0x7f92636287d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92684b2360, local_datums:0x7f92684b2360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:40.664985] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.664987] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.664989] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:40.664991] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.665018] ERROR issue_dba_error (ob_log.cpp:1875) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.665020] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.664994] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b863a0a0, is_inited:true, wash_score:1727337640626660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:499, tablet_addr:[816017-1849-0-2305843639502073856-2](offset=1929216,size=587,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101015}, data_tablet_id:{id:154}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726074677203981000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:693685402, row_count:578}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863a7a0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844390047592448-0](offset=1912832,size=1286,type=2,seq=0), type:"MAJOR", tablet_id:{id:101015}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726074015354550000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843987394409734-0](offset=1914118,size=911,type=2,seq=0), type:"MINOR", tablet_id:{id:101015}, scn_range:{start_scn:{val:1726074677203981000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726205802467242000, upper_trans_version:1726205802467242000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337640626783, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd57f0, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209508621}, this:0x7f92a3bd57f0, timestamp:1727334209508621, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:68, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277803107894000, v:0}, rec_scn:{val:1726241804194470000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771638272-2](offset=1916928,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161786701-0](offset=1917261,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881263599-0](offset=1917423,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc7730, 0x7f92a3bd57f0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.665031] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=37] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.665027] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337640626623, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.665035] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.665042] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640664944, end_time=1727337640665042, runtime=98, *this={this:0x7f92729263c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C664-0-0, dag_ret:0, dag_status:2, start_time:1727337640663453, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.665050] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1597, dag_cnt=13, dag_cnts_[dag.get_type()]=13, &dag=0x7f9264d80630, dag={ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C664-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640663453, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.665056] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C664-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640663444, task_id:YB42C0A8002D-0006230056D8C664-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.665080] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.665084] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.665085] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:40.665089] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.665091] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.665096] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.665104] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640665103, task_id:YB42C0A8002D-0006230056D8C66A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.665106] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=2] schedule one task(task=0x7f9264d82990, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.665119] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.665121] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.665123] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:40.665125] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.665128] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b863eec0, is_inited:true, wash_score:1727337640626660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:499, tablet_addr:[816017-1849-0-2305843639502065664-2](offset=1921024,size=587,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101016}, data_tablet_id:{id:154}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726074677203981000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:398898191, row_count:578}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844419038609408-0](offset=1900544,size=1313,type=2,seq=0), type:"MAJOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726074015354550000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844024975361313-0](offset=1901857,size=946,type=2,seq=0), type:"MINOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:1726074677203981000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726205802467242000, upper_trans_version:1726205802467242000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337640626800, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd5fc0, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209508808}, this:0x7f92a3bd5fc0, timestamp:1727334209508808, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:69, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277803107894000, v:0}, rec_scn:{val:1726241804194470000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771625984-2](offset=1904640,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161774413-0](offset=1904973,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881251311-0](offset=1905135,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc7f00, 0x7f92a3bd5fc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.665162] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=34] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.665166] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.665400] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337640626783, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.665408] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.665427] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=16] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.665430] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.665439] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=6] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101015}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.665470] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=27] succeed to build merge ctx(tablet_id={id:101015}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101015}, table_count:1, [{i:0, table_key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640664982, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=455us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.665493] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d820a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C669-0-0, dag_ret:0, dag_status:2, start_time:1727337640664933, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1490, output_block_cnt_arr:0x7f92728f1498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337640664982, estimated_finish_time:1727337657665493}) [2024-09-26 16:00:40.665499] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d82460, type:15, status:2, dag:{ObIDag:{this:0x7f9264d820a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C669-0-0, dag_ret:0, dag_status:2, start_time:1727337640664933, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.665503] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=4] task finish process(ret=0, start_time=1727337640664981, end_time=1727337640665503, runtime=522, *this={this:0x7f9264d82460, type:15, status:2, dag:{ObIDag:{this:0x7f9264d820a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C669-0-0, dag_ret:0, dag_status:2, start_time:1727337640664933, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.665521] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.665535] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337640626686, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.665528] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752511928, storage_schema:{this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.665543] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753355744, storage_schema:{this:0x7f92728f9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753355744, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.665548] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=15] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.665552] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.665553] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.665554] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.665558] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:456}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753355744, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.665559] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:121}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752511928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.665563] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337640626669, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.665577] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=14] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752725472, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752725472, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.665573] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:456}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753355744, storage_schema:{this:0x7f92728f9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:456}, table_count:1, [{i:0, table_key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8634c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640664942, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=615us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.665591] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.665578] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:121}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640664496, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.06ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.665595] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.665603] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C665-0-0, dag_ret:0, dag_status:2, start_time:1727337640664179, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f92728fb480, output_block_cnt_arr:0x7f92728fb488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337640664496, estimated_finish_time:1727337657665603}) [2024-09-26 16:00:40.665600] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:215}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752725472, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.665611] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d80f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C665-0-0, dag_ret:0, dag_status:2, start_time:1727337640664179, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:40.665616] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=5] task finish process(ret=0, start_time=1727337640664494, end_time=1727337640665616, runtime=1122, *this={this:0x7f9264d80f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C665-0-0, dag_ret:0, dag_status:2, start_time:1727337640664179, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:40.665623] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C667-0-0, dag_ret:0, dag_status:2, start_time:1727337640664920, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f9490, output_block_cnt_arr:0x7f92728f9498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337640664942, estimated_finish_time:1727337657665623}) [2024-09-26 16:00:40.665629] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d81980, type:15, status:2, dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C667-0-0, dag_ret:0, dag_status:2, start_time:1727337640664920, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.665634] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=5] task finish process(ret=0, start_time=1727337640664941, end_time=1727337640665634, runtime=693, *this={this:0x7f9264d81980, type:15, status:2, dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C667-0-0, dag_ret:0, dag_status:2, start_time:1727337640664920, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.665617] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:215}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752725472, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:215}, table_count:1, [{i:0, table_key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640664829, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=769us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.665656] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=33] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C666-0-0, dag_ret:0, dag_status:2, start_time:1727337640664187, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab460, output_block_cnt_arr:0x7f92684ab468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337640664829, estimated_finish_time:1727337657665656}) [2024-09-26 16:00:40.665664] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d81450, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C666-0-0, dag_ret:0, dag_status:2, start_time:1727337640664187, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.665669] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=5] task finish process(ret=0, start_time=1727337640664828, end_time=1727337640665669, runtime=841, *this={this:0x7f9264d81450, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C666-0-0, dag_ret:0, dag_status:2, start_time:1727337640664187, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.665699] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337640626753, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.665708] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.665717] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.665719] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.665723] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:154}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.665738] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:154}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:154}, table_count:1, [{i:0, table_key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8623ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640665077, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=645us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.665756] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C668-0-0, dag_ret:0, dag_status:2, start_time:1727337640664928, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3480, output_block_cnt_arr:0x7f9287db3488, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337640665077, estimated_finish_time:1727337657665755}) [2024-09-26 16:00:40.665760] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=4] succeed to init merge ctx(task={this:0x7f9264d81eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C668-0-0, dag_ret:0, dag_status:2, start_time:1727337640664928, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.665765] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=5] task finish process(ret=0, start_time=1727337640665075, end_time=1727337640665765, runtime=690, *this={this:0x7f9264d81eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C668-0-0, dag_ret:0, dag_status:2, start_time:1727337640664928, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.665808] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101015, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d34060, count:6, allocator:0x7f92864e07d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9287d34390, local_datums:0x7f9287d34390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:40.665905] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.665908] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.665918] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337640626783, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.665957] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640665781, end_time=1727337640665956, runtime=175, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d820a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C669-0-0, dag_ret:0, dag_status:2, start_time:1727337640664933, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.665967] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1033, dag_cnt=12, dag_cnts_[dag.get_type()]=12, &dag=0x7f9264d820a0, dag={ObIDag:{this:0x7f9264d820a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C669-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640664933, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.665975] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C669-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640664931, task_id:YB42C0A8002D-0006230056D8C669-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.666006] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337640626800, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.666014] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92728f7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.666025] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.666027] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.666034] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101016}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.666061] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=23] succeed to build merge ctx(tablet_id={id:101016}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92728f7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101016}, table_count:1, [{i:0, table_key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640665116, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=916us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.666083] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d825d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66A-0-0, dag_ret:0, dag_status:2, start_time:1727337640665106, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f7490, output_block_cnt_arr:0x7f92728f7498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337640665116, estimated_finish_time:1727337657666083}) [2024-09-26 16:00:40.666088] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d82990, type:15, status:2, dag:{ObIDag:{this:0x7f9264d825d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66A-0-0, dag_ret:0, dag_status:2, start_time:1727337640665106, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.666093] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=5] task finish process(ret=0, start_time=1727337640665115, end_time=1727337640666093, runtime=978, *this={this:0x7f9264d82990, type:15, status:2, dag:{ObIDag:{this:0x7f9264d825d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66A-0-0, dag_ret:0, dag_status:2, start_time:1727337640665106, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.666111] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=23] schedule one task(task=0x7f92658d4030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.666117] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=3] schedule one task(task=0x7f9268486030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.666120] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=2] schedule one task(task=0x7f92658c4030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.666124] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=1] schedule one task(task=0x7f92684863c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.666127] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=2] schedule one task(task=0x7f92658d43c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.666131] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337640666129, task_id:YB42C0A8002D-0006230056D8C66B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.666134] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=3] schedule one task(task=0x7f9264d82ec0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.666167] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=215, range=0x7f92a98d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:6}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a64060, count:3, allocator:0x7f92a31fa7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9264a64320, local_datums:0x7f9264a64320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:40.666178] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=121, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:5, allocator:0x7f926368e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3e360, local_datums:0x7f9263a3e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:40.666192] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.666197] INFO [STORAGE] dump_info (ob_sstable_struct.cpp:262) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=14] dump merge info(msg="macro block builder close", output_row_per_s=0, new_macro_KB_per_s=0, *this={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:54, merge_start_time:1727337640666142, merge_finish_time:1727337640666196, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:0, max:0, sum:0, count:0}}, {type:"cost_time", info:{min:0, max:0, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:0, max:0, sum:0, count:0}}, {type:"incremental_row_count", info:{min:0, max:0, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:0, snapshot_version:0, start_scn:0, end_scn:0}, macro_id_list:"", comment:""}) [2024-09-26 16:00:40.666211] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=13] partition merge iter row count(i=0, row_count=0, ghost_row_count=0, pkey={tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, table={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337640626647, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.666224] ERROR issue_dba_error (ob_log.cpp:1875) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.666223] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1544) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=10] merge macro blocks ok(idx_=0, task={this:0x7f92658d4030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C665-0-0, dag_ret:0, dag_status:2, start_time:1727337640664179, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:40.666226] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.666234] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=6] task finish process(ret=0, start_time=1727337640666137, end_time=1727337640666233, runtime=96, *this={this:0x7f92658d4030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C665-0-0, dag_ret:0, dag_status:2, start_time:1727337640664179, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:40.666235] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=0] partition merge iter row count(i=0, row_count=9, ghost_row_count=0, pkey={tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337640626669, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.666264] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640666144, end_time=1727337640666264, runtime=120, *this={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C666-0-0, dag_ret:0, dag_status:2, start_time:1727337640664187, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.666272] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2085, dag_cnt=11, dag_cnts_[dag.get_type()]=11, &dag=0x7f9264d81090, dag={ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C666-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640664187, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.666279] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C666-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640664184, task_id:YB42C0A8002D-0006230056D8C666-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.666281] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101016, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a68060, count:6, allocator:0x7f92864407d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9264a68390, local_datums:0x7f9264a68390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:40.666339] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=20][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.666347] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=154, range=0x7f92aac56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265a80060, count:5, allocator:0x7f926368e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9265a80360, local_datums:0x7f9265a80360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:40.666375] ERROR issue_dba_error (ob_log.cpp:1875) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.666378] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.666388] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337640626800, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.666406] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640666255, end_time=1727337640666406, runtime=151, *this={this:0x7f92658d43c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d825d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66A-0-0, dag_ret:0, dag_status:2, start_time:1727337640665106, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.666413] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.666414] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1308, dag_cnt=10, dag_cnts_[dag.get_type()]=10, &dag=0x7f9264d825d0, dag={ObIDag:{this:0x7f9264d825d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640665106, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.666423] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66A-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337640665103, task_id:YB42C0A8002D-0006230056D8C66A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.666444] ERROR issue_dba_error (ob_log.cpp:1875) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.666446] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.666456] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=1] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337640626753, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.666473] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640666308, end_time=1727337640666473, runtime=165, *this={this:0x7f92684863c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C668-0-0, dag_ret:0, dag_status:2, start_time:1727337640664928, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.666470] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=456, range=0x7f92a9a56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685e6060, count:7, allocator:0x7f92636967d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f92685e63b0, local_datums:0x7f92685e63b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:40.666481] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1552, dag_cnt=9, dag_cnts_[dag.get_type()]=9, &dag=0x7f9264d81af0, dag={ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C668-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640664928, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.666489] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C668-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337640664925, task_id:YB42C0A8002D-0006230056D8C668-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.666515] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.666518] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.666521] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:40.666525] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.666529] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.666531] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.666537] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.666550] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=2] schedule one task(task=0x7f92658d4250, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.666580] ERROR issue_dba_error (ob_log.cpp:1875) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.666582] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=28] succeed to add sys task(task={start_time:1727337640666580, task_id:YB42C0A8002D-0006230056D8C66C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.666585] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=3] schedule one task(task=0x7f9264d833f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.666590] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640666590, task_id:YB42C0A8002D-0006230056D8C66D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.666593] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=2] schedule one task(task=0x7f9264d83920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.666596] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337640666595, task_id:YB42C0A8002D-0006230056D8C66E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.666583] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.666598] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=2] schedule one task(task=0x7f9264d83e50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.666601] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=0] partition merge iter row count(i=0, row_count=45, ghost_row_count=0, pkey={tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337640626686, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.666611] INFO [STORAGE.COMPACTION] get_merged_sstable (ob_tablet_merge_task.cpp:1285) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=4] create new merged sstable(ctx.param_.tablet_id_={id:121}, snapshot_version=1726241132764173999, ctx.param_.merge_type_=3, ctx.create_snapshot_version_=0, table_mode_flag=0) [2024-09-26 16:00:40.666619] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640666444, end_time=1727337640666619, runtime=175, *this={this:0x7f92658c4030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C667-0-0, dag_ret:0, dag_status:2, start_time:1727337640664920, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.666618] INFO [STORAGE] close (ob_index_block_builder.cpp:815) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=3] succeed to close sstable index builder(res={root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_block_ids_.count():0, other_block_ids_.count():0, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, data_column_cnt:10, data_column_checksums:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], row_count:0, max_merged_trans_version:0, contain_uncommitted_row:false, occupy_size:0, original_size:0, data_checksum:0, use_old_macro_block_count:0, compressor_type:1, root_row_store_type:0, nested_offset:0, nested_size:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}) [2024-09-26 16:00:40.666628] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1708, dag_cnt=8, dag_cnts_[dag.get_type()]=8, &dag=0x7f9264d815c0, dag={ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C667-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640664920, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.666635] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C667-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640664916, task_id:YB42C0A8002D-0006230056D8C667-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.666631] INFO [STORAGE] init (ob_sstable.cpp:118) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=8] succeeded to init sstable(ret=0, this={ObITable:{this:0x7f92728fa128, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f92728fa128, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:null, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:40.666670] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.666674] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.666676] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:40.666680] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.666660] INFO [STORAGE.COMPACTION] create_sstable (ob_tablet_merge_ctx.cpp:367) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=28] succeed to merge sstable(param={table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_logic_seq:0, schema_version:1710828752511928, create_snapshot_version:0, progressive_merge_round:0, progressive_merge_step:0, is_ready_for_read:true, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, index_type:0, root_block_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), root_block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, root_row_store_type:0, latest_row_store_type:1, data_index_tree_height:0, data_block_macro_meta_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), data_block_macro_meta:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, use_old_macro_block_count:0, row_count:0, rowkey_column_cnt:5, column_cnt:10, column_checksums:[], data_checksum:0, occupy_size:0, original_size:0, max_merged_trans_version:1726241132764173999, ddl_scn:{val:0, v:0}, contain_uncommitted_row:false, is_meta_root:false, compressor_type:1, encrypt_id:0, master_key_id:0, recycle_version:0, nested_offset:0, nested_size:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}, table_key={tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_merge_info={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:1700, merge_start_time:1727337640664496, merge_finish_time:1727337640666196, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"cost_time", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"incremental_row_count", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:1, snapshot_version:0, start_scn:1726211867305630000, end_scn:1726241548806527000}, macro_id_list:"", comment:""}) [2024-09-26 16:00:40.666683] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8652d40, is_inited:true, wash_score:1727337640626660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[816017-1849-0-2305843644870942720-2](offset=2088960,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:452}, data_tablet_id:{id:452}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2139885871, row_count:290}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b865341c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844508159356928-0](offset=2076672,size=1396,type=2,seq=0), type:"MAJOR", tablet_id:{id:452}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562830808000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843992763282804-0](offset=2078068,size=916,type=2,seq=0), type:"MINI", tablet_id:{id:452}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986149768000, upper_trans_version:1726149986149768000}, {i:1, addr:[816017-1849-0-2305844002426960136-0](offset=2078984,size=925,type=2,seq=0), type:"MINI", tablet_id:{id:452}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726208018712107000, upper_trans_version:1726208018712107000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337640626931, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd9670, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220578716}, this:0x7f92a3bd9670, timestamp:1727334220578716, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:76, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265292959510000, v:0}, rec_scn:{val:1726264902486897000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843480588435456-2](offset=2080768,size=439,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719499703-0](offset=2081207,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881427539-0](offset=2081363,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447947573-0](offset=2081589,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc9e40, 0x7f92a3bd9670, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.666723] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.666724] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=41] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.666726] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.666728] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:40.666731] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.666732] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.666734] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.666740] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.666747] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337640666746, task_id:YB42C0A8002D-0006230056D8C66F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.666750] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=2] schedule one task(task=0x7f9264d84380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.667595] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.667601] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.667605] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:40.667610] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.667614] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.667621] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.667650] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.667653] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.667654] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:40.667657] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.667660] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706339704000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f926e089f20, is_inited:true, wash_score:1727337640626660000, hold_ref_cnt:true, ref_cnt:2, version:2, length:510, tablet_addr:[809012-1445-0-2305843651311779840-2](offset=475136,size=598,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:329}, data_tablet_id:{id:329}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706339704000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:305104969, row_count:10474}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f926e08a620, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844778740674560-0](offset=454656,size=1648,type=2,seq=0), type:"MAJOR", tablet_id:{id:329}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562805980000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843951959471728-0](offset=456304,size=878,type=2,seq=0), type:"MINI", tablet_id:{id:329}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986144852000, upper_trans_version:1726149986144852000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337640627004, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdec60, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221352878}, this:0x7f92a3bdec60, timestamp:1727334221352878, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:87, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265292878107000, v:0}, rec_scn:{val:1726264907253071001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843366770180096-2](offset=458752,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843183160328525-0](offset=459085,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879805423-0](offset=459247,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446325457-0](offset=459473,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bcb5b0, 0x7f92a3bdec60, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:40.667701] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=41] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.667706] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.667841] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337640626931, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.667852] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=15, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.667866] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.667872] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.667877] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:452}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:17, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.667896] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:452}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:452}, table_count:1, [{i:0, table_key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8652d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640666666, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.21ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.667877] INFO [STORAGE] build_new_table_store (ob_tablet_table_store.cpp:1206) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=31] succeed to build new table store(major_tables_={cnt:1, is_inited:true}, minor_tables_={cnt:3, is_inited:true}, memtables_={count:1}, PRINT_TS(*this)={ObTabletTableStore_Pretty:this:0x7f92a9d51ca8, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, memtables:{count:1}, is_ready_for_read:true, table_array:{ [2882] [ YB42C0A8002D-0006230056D8C665-0-0 ] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [2882] [ YB42C0A8002D-0006230056D8C665-0-0 ] MAJOR MAJOR 1725120003420849000 1725120003420849000 {val:0, v:0} {val:1726077600415496000, v:0} 0 false [2882] [ YB42C0A8002D-0006230056D8C665-0-0 ] MINOR MINOR 1726178585269925002 1726178585269925002 {val:1726077259159067000, v:0} {val:1726178585269925002, v:0} 0 false [2882] [ YB42C0A8002D-0006230056D8C665-0-0 ] MINI 1726211867106157000 1726211867106157000 {val:1726178585269925002, v:0} {val:1726211867305630000, v:0} 0 false [2882] [ YB42C0A8002D-0006230056D8C665-0-0 ] MINI 1726241132764173999 1726241132764173999 {val:1726211867305630000, v:0} {val:1726241548806527000, v:0} 0 false [2882] [ YB42C0A8002D-0006230056D8C665-0-0 ] MEM ACTIVE 9223372036854775807 18446744073709547609 {val:1726241548806527000, v:0} {val:4611686018427387903, v:0} 3 unused }}) [2024-09-26 16:00:40.667919] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66C-0-0, dag_ret:0, dag_status:2, start_time:1727337640666585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab480, output_block_cnt_arr:0x7f92684ab488, concurrent_cnt:1, estimate_row_cnt:10, estimate_occupy_size:0, latest_update_ts:1727337640666666, estimated_finish_time:1727337657667919}) [2024-09-26 16:00:40.667926] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d833f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66C-0-0, dag_ret:0, dag_status:2, start_time:1727337640666585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.667931] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=5] task finish process(ret=0, start_time=1727337640666664, end_time=1727337640667931, runtime=1267, *this={this:0x7f9264d833f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66C-0-0, dag_ret:0, dag_status:2, start_time:1727337640666585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.667951] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337640626951, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.667964] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=12] update column cnt(column_cnt_=26, store_column_cnt_=26, column_cnt_=26, column_array_.count()=0) [2024-09-26 16:00:40.667966] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9264b1d480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=26, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.667979] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.667982] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.667987] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:328}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:28, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.668004] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:328}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9264b1d480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:328}, table_count:1, [{i:0, table_key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e072bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640666719, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.27ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.668026] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66D-0-0, dag_ret:0, dag_status:2, start_time:1727337640666592, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d5d0, output_block_cnt_arr:0x7f9264b1d5d8, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337640666719, estimated_finish_time:1727337657668026}) [2024-09-26 16:00:40.668032] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d83920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66D-0-0, dag_ret:0, dag_status:2, start_time:1727337640666592, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.668037] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=5] task finish process(ret=0, start_time=1727337640666718, end_time=1727337640668037, runtime=1319, *this={this:0x7f9264d83920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66D-0-0, dag_ret:0, dag_status:2, start_time:1727337640666592, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.668054] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337640626878, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.668063] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.668072] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.668074] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=1] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.668077] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101063}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.668093] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101063}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101063}, table_count:1, [{i:0, table_key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640666510, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.57ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.668110] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66B-0-0, dag_ret:0, dag_status:2, start_time:1727337640666134, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b4a0, output_block_cnt_arr:0x7f9263a4b4a8, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337640666510, estimated_finish_time:1727337657668110}) [2024-09-26 16:00:40.668115] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d82ec0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66B-0-0, dag_ret:0, dag_status:2, start_time:1727337640666134, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.668119] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=4] task finish process(ret=0, start_time=1727337640666508, end_time=1727337640668119, runtime=1611, *this={this:0x7f9264d82ec0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66B-0-0, dag_ret:0, dag_status:2, start_time:1727337640666134, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.668130] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=2] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.668137] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=4] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.668140] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=2] schedule one task(task=0x7f92665d63c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.668176] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=452, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:15, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:17}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:5, allocator:0x7f92636267d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:17, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,]}) [2024-09-26 16:00:40.668213] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.668258] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.668261] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.668270] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=0] partition merge iter row count(i=0, row_count=10, ghost_row_count=0, pkey={tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337640626931, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.668289] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337640668146, end_time=1727337640668289, runtime=143, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66C-0-0, dag_ret:0, dag_status:2, start_time:1727337640666585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.668297] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1712, dag_cnt=7, dag_cnts_[dag.get_type()]=7, &dag=0x7f9264d83030, dag={ObIDag:{this:0x7f9264d83030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640666585, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.668304] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66C-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640666580, task_id:YB42C0A8002D-0006230056D8C66C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.668378] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337640668374, task_id:YB42C0A8002D-0006230056D8C670-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.668386] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=7] schedule one task(task=0x7f9264d848b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.668430] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.668435] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.668438] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:40.668443] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.668446] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.668397] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337640626968, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.668456] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=58] update column cnt(column_cnt_=12, store_column_cnt_=12, column_cnt_=12, column_array_.count()=0) [2024-09-26 16:00:40.668480] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.668460] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9264b1f490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.668505] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=25] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.668510] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.668524] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=11] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:330}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.668545] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:330}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9264b1f490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:330}, table_count:1, [{i:0, table_key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e107fe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640667588, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=934us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.668576] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66E-0-0, dag_ret:0, dag_status:2, start_time:1727337640666598, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f5f0, output_block_cnt_arr:0x7f9264b1f5f8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337640667588, estimated_finish_time:1727337657668575}) [2024-09-26 16:00:40.668582] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d83e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66E-0-0, dag_ret:0, dag_status:2, start_time:1727337640666598, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.668588] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=5] task finish process(ret=0, start_time=1727337640667586, end_time=1727337640668588, runtime=1002, *this={this:0x7f9264d83e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66E-0-0, dag_ret:0, dag_status:2, start_time:1727337640666598, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.668606] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=4] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.668636] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337640627004, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.668648] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=12] update column cnt(column_cnt_=29, store_column_cnt_=29, column_cnt_=29, column_array_.count()=0) [2024-09-26 16:00:40.668651] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9263a37490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.668661] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.668664] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.668660] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=330, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b2060, count:7, allocator:0x7f92864407d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f92684b23b0, local_datums:0x7f92684b23b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:40.668668] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a37490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:329}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.668685] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:329}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9263a37490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:329}, table_count:1, [{i:0, table_key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e089f20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640667646, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.02ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.668707] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66F-0-0, dag_ret:0, dag_status:2, start_time:1727337640666750, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a375f0, output_block_cnt_arr:0x7f9263a375f8, concurrent_cnt:1, estimate_row_cnt:62, estimate_occupy_size:0, latest_update_ts:1727337640667646, estimated_finish_time:1727337657668707}) [2024-09-26 16:00:40.668712] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d84380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66F-0-0, dag_ret:0, dag_status:2, start_time:1727337640666750, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.668717] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=5] task finish process(ret=0, start_time=1727337640667645, end_time=1727337640668717, runtime=1072, *this={this:0x7f9264d84380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66F-0-0, dag_ret:0, dag_status:2, start_time:1727337640666750, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.668732] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=4] schedule one task(task=0x7f92658c4030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.668785] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=329, range=0x7f92aad52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5a060, count:6, allocator:0x7f9269ffa7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9264d5a390, local_datums:0x7f9264d5a390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:40.668819] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=328, range=0x7f92aa352200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:26, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:28}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a5e060, count:5, allocator:0x7f92ba7f27d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:28, datum_buffer:{capacity:32, datums:0x7f9264a5e360, local_datums:0x7f9264a5e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,]}) [2024-09-26 16:00:40.668845] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.668849] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.668856] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.668859] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=0] partition merge iter row count(i=0, row_count=141, ghost_row_count=0, pkey={tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337640626968, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.668880] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640668630, end_time=1727337640668880, runtime=250, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66E-0-0, dag_ret:0, dag_status:2, start_time:1727337640666598, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.668894] ERROR issue_dba_error (ob_log.cpp:1875) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.668891] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2293, dag_cnt=6, dag_cnts_[dag.get_type()]=6, &dag=0x7f9264d83a90, dag={ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640666598, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.668899] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66E-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337640666595, task_id:YB42C0A8002D-0006230056D8C66E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.668897] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.668907] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=25][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:40.668906] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337640626951, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.668927] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640668765, end_time=1727337640668927, runtime=162, *this={this:0x7f92665d63c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66D-0-0, dag_ret:0, dag_status:2, start_time:1727337640666592, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.668936] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2344, dag_cnt=5, dag_cnts_[dag.get_type()]=5, &dag=0x7f9264d83560, dag={ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640666592, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.668944] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66D-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640666590, task_id:YB42C0A8002D-0006230056D8C66D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.668955] ERROR issue_dba_error (ob_log.cpp:1875) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.668958] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.668965] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:40.668968] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:40.668970] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62070, hex: 0000000000000000, int: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d620a8, hex: 01A3070000000000, int: 500481, num_digit0: 0},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d620e0, hex: 01A3070000000000, int: 500481, num_digit0: 0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62118, hex: 1000000000000000, int: 16},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62150, hex: 901D77210D2D0BE8, int: -1726236497747174000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d62188, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:24321, column_count:31, rowkey_column_count:6, has_column_checksum:0, row_count:47, row_store_type:0, opt:5, var_column_count:0, row_offset:16336, original_length:16464, max_merged_trans_version:1726236510701938000, data_length:16464, data_zlength:16464, data_checksum:1770293370, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f926f604090, buf_size:16464, data_size:16464, row_count:47, column_count:31, max_merged_trans_version:1726236510701938000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1226469540, row_count_delta:47, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16464}) [2024-09-26 16:00:40.668994] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=24][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:40.668997] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:40.668999] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:40.669000] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:40.669002] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:40.669003] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:40.669020] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=101063, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263bea060, count:8, allocator:0x7f92bc9127d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263bea3e0, local_datums:0x7f9263bea3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:40.669085] ERROR issue_dba_error (ob_log.cpp:1875) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.669088] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.669005] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726236497747174000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9264d5a390, local_datums:0x7f9264d5a390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3a0, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3d8, hex: 01A3070000000000, int: 500481, num_digit0: 0},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a410, hex: 01A3070000000000, int: 500481, num_digit0: 0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a448, hex: 1000000000000000, int: 16},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a480, hex: 901D77210D2D0BE8, int: -1726236497747174000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a4b8, hex: 0000000000000000, int: 0},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e0e, hex: 09C6FAC000220600, int: 1726236493268489},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e16, hex: 09C6FAC000220600, int: 1726236493268489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a560, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e1f, hex: E744BCC000220600, int: 1726236489172199},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a5d0, hex: 0200000000000000, int: 2},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a608, hex: 0000000000000000, int: 0},col_id=12:{len: 7, flag: 0, null: 0, ptr: 0x7f9276811e29, hex: 32303030313231, cstr: 2000121},col_id=13:{len: 14, flag: 0, null: 0, ptr: 0x7f9276811e30, hex: 3035303533463030463938393741, cstr: 05053F00F9897A},col_id=14:{len: 7, flag: 0, null: 0, ptr: 0x7f9276811e3e, hex: 31303030313231, cstr: 1000121},col_id=15:{len: 14, flag: 0, null: 0, ptr: 0x7f9276811e45, hex: 3035303533463030423938353344, cstr: 05053F00B9853D},col_id=16:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e53, hex: 0000000000003440, int: 4626322717216342016},col_id=17:{len: 58, flag: 0, null: 0, ptr: 0x7f9276811e5b, hex: 32384235324646443630303030333944303030303230303030303032303030343030333941303032413238363134303839304532333830354230, cstr: 28B52FFD6000039D00002000000200040039A002A286140890E23805B0},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a790, hex: 3A00000000000000, int: 58},col_id=19:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a7c8, hex: A300000000000000, int: 163},col_id=20:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e97, hex: 9F724C16F71F693F, int: 4569218443035570847},col_id=21:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a838, hex: 0200000000000000, int: 2},col_id=22:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a870, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=23:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a8a8, hex: 0000000000000000, int: 0},col_id=24:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a8e0, hex: 0000000000000000, int: 0},col_id=25:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a918, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=26:null,col_id=27:null,col_id=28:null,col_id=29:null,col_id=30:null,]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:329}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828752994696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726236497747174000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9264d5a390, local_datums:0x7f9264d5a390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3a0, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3d8, hex: 01A3070000000000, int: 500481, num_digit0: 0},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a410, hex: 01A3070000000000, int: 500481, num_digit0: 0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a448, hex: 1000000000000000, int: 16},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a480, hex: 901D77210D2D0BE8, int: -1726236497747174000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a4b8, hex: 0000000000000000, int: 0},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e0e, hex: 09C6FAC000220600, int: 1726236493268489},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e16, hex: 09C6FAC000220600, int: 1726236493268489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a560, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e1f, hex: E744BCC000220600, int: 1726236489172199},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a5d0, hex: 0200000000000000, int: 2},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a608, hex: 0000000000000000, int: 0},col_id=12:{len: 7, flag: 0, null: 0, ptr: 0x7f9276811e29, hex: 32303030313231, cstr: 2000121},col_id=13:{len: 14, flag: 0, null: 0, ptr: 0x7f9276811e30, hex: 3035303533463030463938393741, cstr: 05053F00F9897A},col_id=14:{len: 7, flag: 0, null: 0, ptr: 0x7f9276811e3e, hex: 31303030313231, cstr: 1000121},col_id=15:{len: 14, flag: 0, null: 0, ptr: 0x7f9276811e45, hex: 3035303533463030423938353344, cstr: 05053F00B9853D},col_id=16:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e53, hex: 0000000000003440, int: 4626322717216342016},col_id=17:{len: 58, flag: 0, null: 0, ptr: 0x7f9276811e5b, hex: 32384235324646443630303030333944303030303230303030303032303030343030333941303032413238363134303839304532333830354230, cstr: 28B52FFD6000039D00002000000200040039A002A286140890E23805B0},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a790, hex: 3A00000000000000, int: 58},col_id=19:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a7c8, hex: A300000000000000, int: 163},col_id=20:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e97, hex: 9F724C16F71F693F, int: 4569218443035570847},col_id=21:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a838, hex: 0200000000000000, int: 2},col_id=22:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a870, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=23:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a8a8, hex: 0000000000000000, int: 0},col_id=24:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a8e0, hex: 0000000000000000, int: 0},col_id=25:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a918, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=26:null,col_id=27:null,col_id=28:null,col_id=29:null,col_id=30:null,]}, store_ctx:{this:0x7f9269ffa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269ffa118, limit_param:null, stmt_allocator:0x7f9269ffa7d8, allocator:0x7f9269ffa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4919131752989213764, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726236497747174000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9264d5a390, local_datums:0x7f9264d5a390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3a0, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3d8, hex: 01A3070000000000, int: 500481, num_digit0: 0},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a410, hex: 01A3070000000000, int: 500481, num_digit0: 0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a448, hex: 1000000000000000, int: 16},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a480, hex: 901D77210D2D0BE8, int: -1726236497747174000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a4b8, hex: 0000000000000000, int: 0},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e0e, hex: 09C6FAC000220600, int: 1726236493268489},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e16, hex: 09C6FAC000220600, int: 1726236493268489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a560, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e1f, hex: E744BCC000220600, int: 1726236489172199},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a5d0, hex: 0200000000000000, int: 2},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a608, hex: 0000000000000000, int: 0},col_id=12:{len: 7, flag: 0, null: 0, ptr: 0x7f9276811e29, hex: 32303030313231, cstr: 2000121},col_id=13:{len: 14, flag: 0, null: 0, ptr: 0x7f9276811e30, hex: 3035303533463030463938393741, cstr: 05053F00F9897A},col_id=14:{len: 7, flag: 0, null: 0, ptr: 0x7f9276811e3e, hex: 31303030313231, cstr: 1000121},col_id=15:{len: 14, flag: 0, null: 0, ptr: 0x7f9276811e45, hex: 3035303533463030423938353344, cstr: 05053F00B9853D},col_id=16:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e53, hex: 0000000000003440, int: 4626322717216342016},col_id=17:{len: 58, flag: 0, null: 0, ptr: 0x7f9276811e5b, hex: 32384235324646443630303030333944303030303230303030303032303030343030333941303032413238363134303839304532333830354230, cstr: 28B52FFD6000039D00002000000200040039A002A286140890E23805B0},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a790, hex: 3A00000000000000, int: 58},col_id=19:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a7c8, hex: A300000000000000, int: 163},col_id=20:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e97, hex: 9F724C16F71F693F, int: 4569218443035570847},col_id=21:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a838, hex: 0200000000000000, int: 2},col_id=22:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a870, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=23:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a8a8, hex: 0000000000000000, int: 0},col_id=24:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a8e0, hex: 0000000000000000, int: 0},col_id=25:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a918, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=26:null,col_id=27:null,col_id=28:null,col_id=29:null,col_id=30:null,]}, key:{"BIGINT":0},{"BIGINT":500481},{"BIGINT":500481},{"BIGINT":16}, value_iter:{value:{this=0x7f92768119b8 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f9276811d90 latest_compact_node=(nil) max_trans_version={val:1726236497747174000, v:0} max_trans_id=125543295 max_elr_trans_version={val:1726236497747174000, v:0} max_elr_trans_id=125543295 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726236497747174000, v:0} min_modify_scn={val:1726236497747174000, v:0}}, version_iter:this=0x7f9276811a48 trans_version={val:1726236497747174000, v:0} scn={val:1726236497747174000, v:0} tx_id={txid:125543295} prev=(nil) next=0x7f9276811d90 modify_count=0 acc_checksum=1387648185 version=1726236163111105 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:73} seq_no=1726236493032885, multi_version_iter:NULL, max_committed_trans_version:1726236497747174000, cur_trans_version:{val:1726236497747174000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:47, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337640627004, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:31, cur_pos:0, count:0}}) [2024-09-26 16:00:40.669099] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=94][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:40.669096] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=0] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337640626878, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.669114] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640668968, end_time=1727337640669114, runtime=146, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66B-0-0, dag_ret:0, dag_status:2, start_time:1727337640666134, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.669122] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2988, dag_cnt=4, dag_cnts_[dag.get_type()]=4, &dag=0x7f9264d82b00, dag={ObIDag:{this:0x7f9264d82b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640666134, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.669129] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640666129, task_id:YB42C0A8002D-0006230056D8C66B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.669101] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:329}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828752994696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726236497747174000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9264d5a390, local_datums:0x7f9264d5a390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3a0, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3d8, hex: 01A3070000000000, int: 500481, num_digit0: 0},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a410, hex: 01A3070000000000, int: 500481, num_digit0: 0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a448, hex: 1000000000000000, int: 16},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a480, hex: 901D77210D2D0BE8, int: -1726236497747174000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a4b8, hex: 0000000000000000, int: 0},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e0e, hex: 09C6FAC000220600, int: 1726236493268489},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e16, hex: 09C6FAC000220600, int: 1726236493268489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a560, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e1f, hex: E744BCC000220600, int: 1726236489172199},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a5d0, hex: 0200000000000000, int: 2},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a608, hex: 0000000000000000, int: 0},col_id=12:{len: 7, flag: 0, null: 0, ptr: 0x7f9276811e29, hex: 32303030313231, cstr: 2000121},col_id=13:{len: 14, flag: 0, null: 0, ptr: 0x7f9276811e30, hex: 3035303533463030463938393741, cstr: 05053F00F9897A},col_id=14:{len: 7, flag: 0, null: 0, ptr: 0x7f9276811e3e, hex: 31303030313231, cstr: 1000121},col_id=15:{len: 14, flag: 0, null: 0, ptr: 0x7f9276811e45, hex: 3035303533463030423938353344, cstr: 05053F00B9853D},col_id=16:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e53, hex: 0000000000003440, int: 4626322717216342016},col_id=17:{len: 58, flag: 0, null: 0, ptr: 0x7f9276811e5b, hex: 32384235324646443630303030333944303030303230303030303032303030343030333941303032413238363134303839304532333830354230, cstr: 28B52FFD6000039D00002000000200040039A002A286140890E23805B0},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a790, hex: 3A00000000000000, int: 58},col_id=19:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a7c8, hex: A300000000000000, int: 163},col_id=20:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e97, hex: 9F724C16F71F693F, int: 4569218443035570847},col_id=21:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a838, hex: 0200000000000000, int: 2},col_id=22:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a870, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=23:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a8a8, hex: 0000000000000000, int: 0},col_id=24:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a8e0, hex: 0000000000000000, int: 0},col_id=25:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a918, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=26:null,col_id=27:null,col_id=28:null,col_id=29:null,col_id=30:null,]}, store_ctx:{this:0x7f9269ffa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269ffa118, limit_param:null, stmt_allocator:0x7f9269ffa7d8, allocator:0x7f9269ffa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4919131752989213764, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726236497747174000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9264d5a390, local_datums:0x7f9264d5a390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3a0, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a3d8, hex: 01A3070000000000, int: 500481, num_digit0: 0},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a410, hex: 01A3070000000000, int: 500481, num_digit0: 0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a448, hex: 1000000000000000, int: 16},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a480, hex: 901D77210D2D0BE8, int: -1726236497747174000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a4b8, hex: 0000000000000000, int: 0},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e0e, hex: 09C6FAC000220600, int: 1726236493268489},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e16, hex: 09C6FAC000220600, int: 1726236493268489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a560, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e1f, hex: E744BCC000220600, int: 1726236489172199},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a5d0, hex: 0200000000000000, int: 2},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a608, hex: 0000000000000000, int: 0},col_id=12:{len: 7, flag: 0, null: 0, ptr: 0x7f9276811e29, hex: 32303030313231, cstr: 2000121},col_id=13:{len: 14, flag: 0, null: 0, ptr: 0x7f9276811e30, hex: 3035303533463030463938393741, cstr: 05053F00F9897A},col_id=14:{len: 7, flag: 0, null: 0, ptr: 0x7f9276811e3e, hex: 31303030313231, cstr: 1000121},col_id=15:{len: 14, flag: 0, null: 0, ptr: 0x7f9276811e45, hex: 3035303533463030423938353344, cstr: 05053F00B9853D},col_id=16:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e53, hex: 0000000000003440, int: 4626322717216342016},col_id=17:{len: 58, flag: 0, null: 0, ptr: 0x7f9276811e5b, hex: 32384235324646443630303030333944303030303230303030303032303030343030333941303032413238363134303839304532333830354230, cstr: 28B52FFD6000039D00002000000200040039A002A286140890E23805B0},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a790, hex: 3A00000000000000, int: 58},col_id=19:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a7c8, hex: A300000000000000, int: 163},col_id=20:{len: 8, flag: 0, null: 0, ptr: 0x7f9276811e97, hex: 9F724C16F71F693F, int: 4569218443035570847},col_id=21:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a838, hex: 0200000000000000, int: 2},col_id=22:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a870, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=23:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a8a8, hex: 0000000000000000, int: 0},col_id=24:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a8e0, hex: 0000000000000000, int: 0},col_id=25:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5a918, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=26:null,col_id=27:null,col_id=28:null,col_id=29:null,col_id=30:null,]}, key:{"BIGINT":0},{"BIGINT":500481},{"BIGINT":500481},{"BIGINT":16}, value_iter:{value:{this=0x7f92768119b8 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f9276811d90 latest_compact_node=(nil) max_trans_version={val:1726236497747174000, v:0} max_trans_id=125543295 max_elr_trans_version={val:1726236497747174000, v:0} max_elr_trans_id=125543295 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726236497747174000, v:0} min_modify_scn={val:1726236497747174000, v:0}}, version_iter:this=0x7f9276811a48 trans_version={val:1726236497747174000, v:0} scn={val:1726236497747174000, v:0} tx_id={txid:125543295} prev=(nil) next=0x7f9276811d90 modify_count=0 acc_checksum=1387648185 version=1726236163111105 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:73} seq_no=1726236493032885, multi_version_iter:NULL, max_committed_trans_version:1726236497747174000, cur_trans_version:{val:1726236497747174000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:47, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337640627004, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:31, cur_pos:0, count:0}}]) [2024-09-26 16:00:40.669149] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=48][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:40.669152] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=2] partition merge iter row count(i=0, row_count=47, ghost_row_count=0, pkey={tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337640627004, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.669163] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:40.669171] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:40.669176] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=5][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:40.669178] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337640668744, end_time=1727337640669178, runtime=434, *this={this:0x7f92658c4030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66F-0-0, dag_ret:0, dag_status:2, start_time:1727337640666750, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.669185] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f92658c4030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66F-0-0, dag_ret:0, dag_status:2, start_time:1727337640666750, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:40.669193] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=2443, dag_cnt=3, dag_cnts_[dag.get_type()]=3, &dag=0x7f9264d83fc0, dag={ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C66F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640666750, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.669201] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C66F-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640666746, task_id:YB42C0A8002D-0006230056D8C66F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.669242] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337640627025, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.669254] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=25, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.669267] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.669270] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.669275] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:332}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:27, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.669295] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:332}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:332}, table_count:1, [{i:0, table_key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8648b00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640668411, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=862us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.669318] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C670-0-0, dag_ret:0, dag_status:2, start_time:1727337640668385, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab490, output_block_cnt_arr:0x7f92684ab498, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337640668411, estimated_finish_time:1727337657669318}) [2024-09-26 16:00:40.669324] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d848b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C670-0-0, dag_ret:0, dag_status:2, start_time:1727337640668385, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.669329] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=5] task finish process(ret=0, start_time=1727337640668407, end_time=1727337640669329, runtime=922, *this={this:0x7f9264d848b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C670-0-0, dag_ret:0, dag_status:2, start_time:1727337640668385, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.669344] INFO [MDS] init_for_evict_medium_info (ob_tablet_mds_data.cpp:414) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=37] succeeded to init mds data(ret=0, finish_medium_scn=1726077600415496000, src_medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, src_extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}) [2024-09-26 16:00:40.669372] INFO [STORAGE] init_for_merge (ob_tablet.cpp:363) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=18] succeeded to init tablet for mini/minor/major merge(ret=0, param={sstable:0x7f92728fa128, snapshot_version:1726241132764173999, clog_checkpoint_scn:{val:1726241548806527000, v:0}, multi_version_start:1726178585269925002, need_report:false, storage_schema:{this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, rebuild_seq:0, update_with_major_flag:false, need_check_sstable:false, ddl_info:{keep_old_ddl_sstable:true, ddl_start_scn:{val:0, v:0}, ddl_commit_scn:{val:0, v:0}, ddl_checkpoint_scn:{val:0, v:0}, ddl_snapshot_version:0, ddl_execution_id:-1, data_format_version:0}, allow_duplicate_sstable:false, merge_type:"MINI_MERGE", need_check_transfer_seq:true, transfer_seq:0}, old_tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337640625574000, hold_ref_cnt:true, ref_cnt:3, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337640626647, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}, this={this:0x7f9263a42060, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x7f92a9d52450, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726241548806527000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726241132764173999, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f9263a42670, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}, {i:2, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726241132764173999, upper_trans_version:1726241132764173999}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x7f9263a42520, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9263a43880, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9263a43950, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{redefined:false, snapshot_version:0, schema_version:0, data_tablet_id:{id:0}, hidden_tablet_id:{id:0}, lob_meta_tablet_id:{id:50121}, lob_piece_tablet_id:{id:60121}, is_old_mds:true}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:40.669477] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=78] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9263a4a060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}}, ref_cnt:0, upper_trans_version:1725120003420849000, timestamp:0}, this:0x7f9263a4a060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1725120003420849000, max_merged_trans_version:1725120003420849000, data_macro_block_count:1, nested_size:4096, nested_offset:589824, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:84, row_count_:14, occupy_size_:1604, original_size_:1914, data_checksum_:3587602083, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:1, data_micro_block_count_:1, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:2, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1725120003420849000, max_merged_trans_version_:1725120003420849000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9263a4a310, column_checksum_count:10, data_root_info:{addr:[0-0-0-3458764933653594112-0](offset=0,size=391,type=3,seq=0), block_data:{buf:0x7f9263a4a360, size:391, extra_buf:0x7f9263a4a4e7, extra_size:424, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-3458765009889263616-0](offset=0,size=462,type=3,seq=0), block_data:{buf:0x7f9263a4a68f, size:462, extra_buf:null, extra_size:0, type:1}}, data_block_count:1, other_block_count:0, linked_block_count:0, data_block_ids:0x7f9263a4a85d, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:true, nested_offset:589824, nested_size:4096}}, valid_for_reading:true}) [2024-09-26 16:00:40.669497] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=18] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f92728f2060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}}, ref_cnt:0, upper_trans_version:1726178585269925002, timestamp:0}, this:0x7f92728f2060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726178585269925002, max_merged_trans_version:1726178585269925002, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726178585269925002, v:0}, meta:{basic_meta:{version:1, length:94, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726178585269925002, max_merged_trans_version_:1726178585269925002, recycle_version:1726077600415496000, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726178585269925002, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:1, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f92728f2310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:40.669508] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=11] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9263a4a060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}}, ref_cnt:0, upper_trans_version:1726211867106157000, timestamp:0}, this:0x7f9263a4a060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726211867106157000, max_merged_trans_version:1726211867106157000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726211867305630000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726211867106157000, max_merged_trans_version_:1726211867106157000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726211867305630000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9263a4a310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:40.669518] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=9] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f92728f2060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f92728f2060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f92728f2310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:40.669529] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=9][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.669561] ERROR issue_dba_error (ob_log.cpp:1875) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.669564] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf9de5b1 0xf9dcfd6 0xe51c6fb 0xe519c03 0xe518f89 0xe516f8c 0xf01158d 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.669582] INFO [STORAGE] reset (ob_tablet.cpp:159) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=0] reset tablet(this=0x7f9263a42060, ls_id={id:1}, tablet_id={id:121}, lbt()="0x121f1f9c 0xe52cc09 0xe52c6c5 0xe52bd13 0x4fd348f 0xf01110c 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:40.669600] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1250) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=0][errcode=-4184] sstable merge finish(ret=-4184, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C665-0-0, dag_ret:0, dag_status:2, start_time:1727337640664179, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f92728fb480, output_block_cnt_arr:0x7f92728fb488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337640664496, estimated_finish_time:1727337657665603}, compaction_filter:NULL, time_guard:total=2.20ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, task={this:0x7f92658d4250, type:16, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C665-0-0, dag_ret:0, dag_status:2, start_time:1727337640664179, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:40.669625] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640666609, end_time=1727337640669625, runtime=3016, *this={this:0x7f92658d4250, type:16, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C665-0-0, dag_ret:0, dag_status:2, start_time:1727337640664179, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:40.669632] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5453, dag_cnt=2, dag_cnts_[dag.get_type()]=2, &dag=0x7f9264d80b60, dag={ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C665-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640664179, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}) [2024-09-26 16:00:40.669639] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C665-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640664175, task_id:YB42C0A8002D-0006230056D8C665-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.669662] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=3] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.669668] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337640669667, task_id:YB42C0A8002D-0006230056D8C671-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.669671] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=3] schedule one task(task=0x7f9264d84de0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.669703] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=332, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:25, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:27}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:6, allocator:0x7f92691a27d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:27, datum_buffer:{capacity:32, datums:0x7f92665f8390, local_datums:0x7f92665f8390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,]}) [2024-09-26 16:00:40.669737] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:40.669768] ERROR issue_dba_error (ob_log.cpp:1875) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.669771] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.669778] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337640627025, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.669795] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640669674, end_time=1727337640669794, runtime=120, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C670-0-0, dag_ret:0, dag_status:2, start_time:1727337640668385, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.669804] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1418, dag_cnt=1, dag_cnts_[dag.get_type()]=1, &dag=0x7f9264d844f0, dag={ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C670-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640668385, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.669810] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C670-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337640668374, task_id:YB42C0A8002D-0006230056D8C670-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.670130] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.670135] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:40.670138] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:40.670143] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:40.670147] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:40.670154] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.670409] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337640627052, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:40.670425] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:40.670468] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:40.670472] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:40.670479] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101067}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:40.670499] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:101067}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101067}, table_count:1, [{i:0, table_key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b86a2d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337640670123, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=353us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:40.670529] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C671-0-0, dag_ret:0, dag_status:2, start_time:1727337640669671, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab4a0, output_block_cnt_arr:0x7f92684ab4a8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337640670123, estimated_finish_time:1727337657670528}) [2024-09-26 16:00:40.670535] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d84de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C671-0-0, dag_ret:0, dag_status:2, start_time:1727337640669671, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.670541] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=5] task finish process(ret=0, start_time=1727337640670121, end_time=1727337640670541, runtime=420, *this={this:0x7f9264d84de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C671-0-0, dag_ret:0, dag_status:2, start_time:1727337640669671, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.670568] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=2] schedule one task(task=0x7f9268486030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:40.670609] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101067, range=0x7f92aa6d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:8, allocator:0x7f92a31fa7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92665f83e0, local_datums:0x7f92665f83e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:40.670819] ERROR issue_dba_error (ob_log.cpp:1875) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:40.670825] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:40.670834] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=0] partition merge iter row count(i=0, row_count=227, ghost_row_count=0, pkey={tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337640627052, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:40.670851] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337640670581, end_time=1727337640670851, runtime=270, *this={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C671-0-0, dag_ret:0, dag_status:2, start_time:1727337640669671, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:40.670859] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1188, dag_cnt=0, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264d84a20, dag={ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C671-0-0, dag_ret:-4184, dag_status:5, start_time:1727337640669671, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:40.670866] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C671-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337640669667, task_id:YB42C0A8002D-0006230056D8C671-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:40.680035] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:40.692740] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:40.692750] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.701592] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:40.701604] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337640701588) [2024-09-26 16:00:40.701610] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337640601058, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:40.701626] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223369938045, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.701632] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:40.701640] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:40.701644] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:40.701655] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:40.701663] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.701665] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.701667] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337640701613) [2024-09-26 16:00:40.709336] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337640312882]}) [2024-09-26 16:00:40.723012] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27] success to notify_need_writing_throttling True [2024-09-26 16:00:40.723020] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.753291] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:40.753301] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.756203] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=5] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:40.783567] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:40.783576] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.785507] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337640785506, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007962, add_bytes: 0 [2024-09-26 16:00:40.802132] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:40.802143] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337640802129) [2024-09-26 16:00:40.802148] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337640701613, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:40.802163] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223370038589, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.802169] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:40.802177] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:40.802181] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:40.802184] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:40.802190] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.802192] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.802193] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337640802151) [2024-09-26 16:00:40.809946] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337640312882]}) [2024-09-26 16:00:40.813834] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:40.813855] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.813961] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:40.813972] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337640813959], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:40.813979] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337640813959], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:40.813983] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:40.813986] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:40.820686] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.820689] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.820698] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.820702] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.820711] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.821647] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.821656] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.821663] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.821666] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.821675] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.822062] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:40.822728] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:40.822832] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:40.823433] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1195366, clean_num=31457) [2024-09-26 16:00:40.844151] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:40.844157] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.853856] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=12][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337640813959]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:40.853871] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=15][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:40.864636] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337641795971, now=[mts=1727337639866208]) [2024-09-26 16:00:40.864647] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337641795971, now=[mts=1727337639866208], now0=[mts=1727337639866208], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:40.864654] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a3123ce0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500738, assoc_session_id:3221500738, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337639864548, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:40.864671] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22fdff8, id:3221500738, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3123ce0}) [2024-09-26 16:00:40.864679] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500738, *tx_desc={this:0x7f92a3123ce0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500738, assoc_session_id:3221500738, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337639864548, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:40.864686] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500738, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3123ce0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500738, assoc_session_id:3221500738, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337639864548, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337639866185, use_das=false, nested_level=0, session={this:0x7f92a22fdff8, id:3221500738, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3123ce0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:40.864703] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:40.864707] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:40.864710] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:40.864713] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:40.864715] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:40.864719] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:40.864723] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=1001248, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337641865971) [2024-09-26 16:00:40.864727] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:40.864734] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:40.864736] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:40.864737] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:40.864743] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:40.864781] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:40.864784] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:40.874205] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:40.874223] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.890035] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=4] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-09-26 16:00:40.890063] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=23] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=688) [2024-09-26 16:00:40.902215] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:40.902227] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337640902211) [2024-09-26 16:00:40.902233] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337640802151, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:40.902239] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:40.902262] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:40.902273] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:40.902618] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885750-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:40.902665] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885750-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:40.902703] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337640902702, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=446, wlock_time=20, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:40.902713] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:40.902729] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223370138235, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:40.902735] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:40.902741] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:40.902749] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:40.902752] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:40.902757] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.902760] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:40.902762] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337640902718) [2024-09-26 16:00:40.904744] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:40.904753] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.910534] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337640813959]}) [2024-09-26 16:00:40.934943] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:40.934953] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.942792] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:83, tid:2892}]) [2024-09-26 16:00:40.956467] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:40.965236] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:40.965246] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.965257] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:40.976103] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:40.993451] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=44] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:40.997528] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:40.997538] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:40.999442] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337639999631]) [2024-09-26 16:00:40.999450] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337639999631], now0=[mts=1727337639999631], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:40.999456] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337639998614, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:40.999485] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=28][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:40.999496] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337639998614, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:40.999504] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337639998614, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:40.999521] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:40.999527] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:40.999531] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:40.999533] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:40.999536] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:40.999539] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=3, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:40.999544] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=3000, remain_us=25994788, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:41.002359] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:41.002365] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:41.002370] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337641002357) [2024-09-26 16:00:41.002376] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337640902236, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:41.002392] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223370237845, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:41.002400] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:41.002407] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:41.002413] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:41.002416] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:41.002421] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.002422] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.002424] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337641002379) [2024-09-26 16:00:41.002599] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:3, local_retry_times:3, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:41.002614] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:41.002617] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:41.002618] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:41.002624] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:41.002662] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:41.002667] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:41.007390] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] new mysql sessid created(conn.sessid_=3221500744, support_ssl=false) [2024-09-26 16:00:41.007422] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221500744, sess.client_addr_="127.0.0.1:36342") [2024-09-26 16:00:41.007427] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500744, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:41.007448] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:41.007471] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2688][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:41.007521] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9E-0-0] [lt=8] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:41.007705] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9E-0-0] [lt=4][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:41.007712] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9E-0-0] [lt=6][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"hzQe)9%I"ku+r"NA;C~s"}, ret=-4043) [2024-09-26 16:00:41.007718] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9E-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:41.007722] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9E-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:41.007729] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9E-0-0] [lt=1] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:41.007749] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9E-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500744, proxy_sessid:0}) [2024-09-26 16:00:41.007753] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9E-0-0] [lt=4] mark session id unused(sessid=3221500744) [2024-09-26 16:00:41.007760] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9E-0-0] [lt=5][errcode=0] server close connection(sessid=3221500744, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:41.007763] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9E-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500744, proxy_sessid=0) [2024-09-26 16:00:41.007765] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9E-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:41.007768] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9E-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500744, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:41.007826] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500744, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337641007253, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:41.007837] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221500744, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:41.007840] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500744, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337641007253, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:41.007844] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500744, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:41.011145] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337640813959]}) [2024-09-26 16:00:41.023292] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=5] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:41.023309] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=15][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-09-26 16:00:41.023314] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:41.023319] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=5][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-09-26 16:00:41.023324] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:41.023327] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:41.023332] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-09-26 16:00:41.023335] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:41.023352] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=16][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:41.023359] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=6][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:41.023362] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=3][errcode=-5019] resolve table failed(ret=-5019) [2024-09-26 16:00:41.023365] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:2822) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=3][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-09-26 16:00:41.023368] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2676) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=2][errcode=-5019] resolve joined table failed(ret=-5019) [2024-09-26 16:00:41.023374] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=5][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:41.023377] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:41.023381] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:41.023383] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:41.023401] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:41.023408] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=6][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:41.023412] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:41.023415] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:41.023418] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-09-26 16:00:41.023422] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=4][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:41.023428] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:41.023435] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:41.023443] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:41.023448] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:41.023451] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=3][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:41.023474] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=18][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:41.023491] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284F9F-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:41.023496] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [3024][BlackListServic][T0][YB42C0A8002D-0006230056284F9F-0-0] [lt=4][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-09-26 16:00:41.023500] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:41.023503] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:41.023506] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:41.023509] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337641023174, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:41.023514] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:41.023517] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:222) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:41.023547] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3] ls blacklist refresh finish(cost_time=710) [2024-09-26 16:00:41.023582] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:41.023682] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:41.024282] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1226823, clean_num=31457) [2024-09-26 16:00:41.028764] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:41.028774] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.029027] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590A-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:41.029037] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590A-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:41.029055] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590A-0-0] [lt=17][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:41.029061] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590A-0-0] [lt=6][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:41.043686] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:41.043698] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:41.043702] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:41.043705] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:41.043708] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:41.043710] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:41.043713] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:41.043715] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:41.043717] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:41.043719] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:41.043720] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:41.043722] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:41.043757] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=34][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:41.043759] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:41.043763] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:41.043770] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=7][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:41.043773] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:41.043775] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:41.043777] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:41.043779] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:41.043781] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:41.043785] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:41.043789] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:41.043791] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:41.043792] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:41.043798] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:41.043801] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:41.043809] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85882-0-0] [lt=7][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:41.043812] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85882-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:41.043814] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:41.043816] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:41.043819] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337641043632, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:41.043824] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85882-0-0] [lt=5][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:41.043827] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:41.043844] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:41.043846] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85882-0-0] [lt=2][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:41.054110] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:41.054440] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:41.054451] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:41.054454] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:41.054475] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=21][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:41.054478] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:41.054484] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:41.054488] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:41.054490] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:41.054491] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:41.054493] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:41.054494] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:41.054496] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:41.054498] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:41.054500] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:41.054504] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:41.054506] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:41.054508] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:41.054509] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:41.054511] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:41.054513] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:41.054515] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:41.054519] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:41.054523] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:41.054524] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:41.054525] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:41.054529] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:41.054532] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E5E-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:41.054534] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2579][RSAsyncTask0][T0][YB42C0A8002D-0006230057584E5E-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:41.054536] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:41.054538] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:41.054540] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:41.054541] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337641054400, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:41.054544] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:41.054599] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:41.057043] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: SHARED_BLOCK_ReaderWriter_IO, mode: read, size: 4096.00, iops: 52.89, rt: 2899.64) [2024-09-26 16:00:41.057051] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:41.057073] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 12190.00) [2024-09-26 16:00:41.057077] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:41.057082] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:41.057096] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:41.058987] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:41.059005] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.084404] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:41.089285] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:41.089293] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.102951] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:41.102968] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337641102948) [2024-09-26 16:00:41.102973] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337641002379, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:41.102989] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223370338469, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:41.102996] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:41.103003] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:41.103008] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:41.103011] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:41.103016] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.103018] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.103019] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337641102976) [2024-09-26 16:00:41.103029] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D8A6-0-0] [lt=10][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337641102963}) [2024-09-26 16:00:41.111748] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337640813959]}) [2024-09-26 16:00:41.119526] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:41.119534] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.142002] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:221) [2596][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=9] [ASYNC TASK QUEUE](queue_size=0) [2024-09-26 16:00:41.149693] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:41.149702] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.157020] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:41.180146] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:41.180155] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.202777] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:41.202790] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=8] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:41.202804] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=13] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:41.202984] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223370439135, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:41.202995] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:41.203003] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:41.203009] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:41.203012] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:41.203018] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.203020] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.203022] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337641202969) [2024-09-26 16:00:41.210392] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:41.210399] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.212381] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337640813959]}) [2024-09-26 16:00:41.225158] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:41.225261] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=50, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:41.225862] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1258280, clean_num=31457) [2024-09-26 16:00:41.240660] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:41.240669] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.241404] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:41.241414] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:41.241431] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=16][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:41.241435] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:41.241439] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:41.241442] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:41.241445] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:41.241449] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:41.241451] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:41.241453] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:41.241454] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:41.241457] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:41.241459] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:41.241479] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=20][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:41.241486] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:41.241489] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:41.241492] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:41.241502] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:41.241506] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:41.241509] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:41.241512] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:41.241519] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:41.241525] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:41.241528] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:41.241529] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:41.241535] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:41.241538] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:41.241540] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:41.241543] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:41.241545] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=1][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:41.241547] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:41.241549] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337641241332, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:41.241552] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:41.241555] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:41.241594] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=3][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:41.241598] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=3][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:41.241600] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:41.241602] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:41.241608] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=5][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:41.241611] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=3][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:41.241613] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485891-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:41.270948] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:41.270974] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.301214] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:41.301223] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.303229] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:41.303240] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337641303226) [2024-09-26 16:00:41.303247] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337641102976, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:41.303275] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=223370539741, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:41.303281] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:41.303289] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:41.303295] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:41.303300] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:41.303306] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.303308] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.303310] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337641303260) [2024-09-26 16:00:41.303737] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337642233354, now=[mts=1727337640303654]) [2024-09-26 16:00:41.303746] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337642233354, now=[mts=1727337640303654], now0=[mts=1727337640303654], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:41.303752] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269158080, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337640303009, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:41.303777] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269158080}) [2024-09-26 16:00:41.303786] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269158080, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337640303009, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:41.303794] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=8] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269158080, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337640303009, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337640303628, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269158080}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:41.303821] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=26][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:41.303827] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:41.303830] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:41.303832] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:41.303834] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:41.303838] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:41.303843] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=999512, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337642303354) [2024-09-26 16:00:41.303846] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:41.303853] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:41.303855] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:41.303857] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:41.303865] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:41.303903] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:41.303906] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:41.312941] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337640813959]}) [2024-09-26 16:00:41.314331] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.314339] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337641314329], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:41.314344] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337641314329], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:41.314348] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=5128, get_gts_with_stc_cnt=11377, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-09-26 16:00:41.314352] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337641314329]}) [2024-09-26 16:00:41.325676] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D3AFE-0-0] [lt=20][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:41.331478] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:41.331487] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.346045] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337641346044, bytes: 14635822, bw: 0.003620 MB/s, add_ts: 1006891, add_bytes: 3822 [2024-09-26 16:00:41.357138] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:41.361614] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:41.361624] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.386189] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=23] current monitor number(seq_=-1) [2024-09-26 16:00:41.386224] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:41.392050] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:41.392058] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.403787] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:41.403800] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337641403784) [2024-09-26 16:00:41.403816] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337641303260, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:41.403829] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:41.403840] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:41.403844] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:41.404165] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:41.404177] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:41.404182] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:41.404185] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:41.404190] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=4][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:41.404192] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:41.404198] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:41.404201] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:41.404203] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:41.404205] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:41.404207] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:41.404210] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:41.404213] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:41.404216] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:41.404223] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:41.404226] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:41.404229] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:41.404233] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:41.404236] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:41.404239] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:41.404241] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:41.404247] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:41.404253] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:41.404256] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:41.404257] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:41.404264] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:41.404267] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:41.404270] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885751-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:41.404274] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:41.404277] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:41.404280] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:41.404282] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337641404086, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:41.404286] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:41.404289] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:41.404318] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:41.404323] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337641404323, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=487, wlock_time=11, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:41.404331] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:41.404336] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:41.404352] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223370640399, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:41.404357] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:41.404363] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:41.404368] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:41.404372] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:41.404378] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.404380] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.404381] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337641404340) [2024-09-26 16:00:41.413560] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337641314329]}) [2024-09-26 16:00:41.422348] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:41.422366] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.427339] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:41.427446] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=57, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:41.428059] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1289737, clean_num=31457) [2024-09-26 16:00:41.452611] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:41.452619] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.454566] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.454574] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.454579] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.455530] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.455541] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.455546] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.455551] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.455538] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.455553] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.455555] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.455556] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.455561] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.456512] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.456516] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.456519] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.456522] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.456523] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.456525] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.456514] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.456531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.456536] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.457500] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.457500] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.457505] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.457509] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.457507] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.457512] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.457518] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.457519] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.457521] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.458455] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.458460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.458463] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.458473] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.458478] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.458481] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.458485] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.458488] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.458489] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.459452] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.459470] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.459473] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.459477] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.459479] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.459480] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.459479] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.459486] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.459490] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:41.460492] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.460538] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.460548] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.461391] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.461390] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.461398] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.462357] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.462366] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.462372] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.463005] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.463310] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.463326] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.463524] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.464332] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.464343] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.464350] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.464939] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.464945] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.465124] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.465461] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.465465] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.466325] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.466331] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.466345] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.466938] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.466942] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.466946] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.468247] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.468257] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.468262] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.469205] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.469227] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.469238] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.470171] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.470178] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.470186] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.470768] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.470792] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.470797] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.472138] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.472157] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.472188] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.473105] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.473153] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.473166] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.473769] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.473813] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.473821] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.475066] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.475067] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.475074] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.476046] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.476052] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.476056] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.476772] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.476809] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.476816] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.478010] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.478016] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.478027] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.478627] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.478631] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.478635] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.479961] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.479967] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.479972] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.480942] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.480960] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.480965] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.481894] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.481895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.481902] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.482932] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.482957] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:41.482969] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.482994] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.482998] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.483900] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.483906] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.483902] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.484884] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.484898] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.484909] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.485808] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.485809] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.485816] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:41.499109] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] ====== tenant freeze timer task ====== [2024-09-26 16:00:41.499118] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:41.499148] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337640625515, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337640625479, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337640625532, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:41.499184] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:41.499245] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D9-0-0] [lt=4] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:41.499254] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D9-0-0] [lt=7] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:41.499260] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D9-0-0] [lt=4] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:41.499263] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D9-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:41.499267] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D9-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:41.499270] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D9-0-0] [lt=3][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:41.499275] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D9-0-0] [lt=4][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:41.499294] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D9-0-0] [lt=19] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:41.499297] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D9-0-0] [lt=3][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:41.499300] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D9-0-0] [lt=3] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:41.499303] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851D9-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:41.499335] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851D9-0-0] [lt=3] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:41.504401] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:41.504413] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337641504398) [2024-09-26 16:00:41.504418] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337641403819, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:41.504434] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223370738943, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:41.504440] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:41.504447] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:41.504452] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:41.504467] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:41.504472] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.504484] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.504486] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337641504421) [2024-09-26 16:00:41.513161] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:41.513179] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.514205] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337641314329]}) [2024-09-26 16:00:41.522806] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590B-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:41.522816] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590B-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:41.522824] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590B-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:41.522830] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590B-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:41.541504] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6431, mem_used=12478464) [2024-09-26 16:00:41.543460] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:41.543468] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.557842] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:41.562978] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:00:41.574752] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:41.574759] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.601592] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.601604] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.601629] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.601628] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.604974] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:41.604990] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337641604971) [2024-09-26 16:00:41.604996] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337641504421, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:41.605001] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:41.605007] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.605012] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223370840517, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:41.605047] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:41.605055] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:41.605069] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:41.605072] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:41.605078] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.605080] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.605082] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337641604999) [2024-09-26 16:00:41.605091] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:41.614740] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337641314329]}) [2024-09-26 16:00:41.628571] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:41.628674] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=51, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:41.629269] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1321194, clean_num=31457) [2024-09-26 16:00:41.635240] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:41.635249] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.649952] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4184, dropped:500, tid:2858}]) [2024-09-26 16:00:41.658638] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=4] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:41.665510] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:41.665519] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.683336] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:41.695681] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:41.695691] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.705632] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:41.705644] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337641705629) [2024-09-26 16:00:41.705650] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337641604999, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:41.705678] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223370942053, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:41.705695] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:41.705703] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:41.705709] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:41.705711] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:41.705717] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.705720] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.705721] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337641705653) [2024-09-26 16:00:41.714889] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337641314329]}) [2024-09-26 16:00:41.725877] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:41.725885] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.756336] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:41.756346] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.758322] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:41.787610] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24] success to notify_need_writing_throttling True [2024-09-26 16:00:41.787663] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=51][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.792507] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337641792506, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007000, add_bytes: 0 [2024-09-26 16:00:41.806199] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:41.806227] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337641806195) [2024-09-26 16:00:41.806235] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337641705653, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:41.806265] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223371042642, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:41.806274] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:41.806284] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:41.806290] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:41.806294] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:41.806300] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.806304] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.806306] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337641806238) [2024-09-26 16:00:41.811090] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [2915][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=7] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-09-26 16:00:41.815005] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337641314329]}) [2024-09-26 16:00:41.815062] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:41.815074] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337641815059], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:41.815082] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337641815059], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:41.815086] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:41.815092] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:41.817733] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:41.817745] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.821883] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.821883] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.821894] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.821899] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.821889] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.821902] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.821908] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.821909] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.821913] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.821921] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.821927] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:41.829745] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:41.829862] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=58, refresh_score_time=26, wash_time=2) [2024-09-26 16:00:41.830477] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1352651, clean_num=31457) [2024-09-26 16:00:41.848164] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:41.848177] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.865449] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=6][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337641795971, now=[mts=1727337640864780]) [2024-09-26 16:00:41.865464] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=14][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337641795971, now=[mts=1727337640864780], now0=[mts=1727337640864780], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:41.865487] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=21][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a3123ce0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500738, assoc_session_id:3221500738, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337640864618, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:41.865506] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=18][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22fdff8, id:3221500738, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3123ce0}) [2024-09-26 16:00:41.865514] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500738, *tx_desc={this:0x7f92a3123ce0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500738, assoc_session_id:3221500738, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337640864618, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:41.865522] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500738, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3123ce0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500738, assoc_session_id:3221500738, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337640864618, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337639866185, use_das=false, nested_level=0, session={this:0x7f92a22fdff8, id:3221500738, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3123ce0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:41.865539] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:41.865544] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:41.865546] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:41.865549] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:41.865551] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:41.865555] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:41.865560] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4] will sleep(sleep_us=411, remain_us=411, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4283, timeout_timestamp=1727337641865971) [2024-09-26 16:00:41.865998] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:41.866012] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=9][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:41.866015] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:41.866016] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:41.866022] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:41.866064] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1764) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4012] get gts timeout(ret=-4012, expire_ts=1727337641795971, current_time=1727337641865442) [2024-09-26 16:00:41.866067] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=3][errcode=-4012] acquire global snapshot fail(ret=-4012, gts_ahead=0, expire_ts=1727337641795971, now=[mts=1727337641866063], now0=[mts=1727337641866063], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:41.866071] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=4][errcode=-4012] acquire global snapshot fail(ret=-4012, tx={this:0x7f92a3123ce0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500738, assoc_session_id:3221500738, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337641865442, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:41.866083] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=11][errcode=-4012] fail to get snapshot(ret=-4012, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22fdff8, id:3221500738, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3123ce0}) [2024-09-26 16:00:41.866089] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=6][errcode=-4012] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4012, session_id=3221500738, *tx_desc={this:0x7f92a3123ce0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500738, assoc_session_id:3221500738, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337641865442, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:41.866094] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=5] start stmt(ret=-4012, auto_commit=true, session_id=3221500738, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3123ce0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500738, assoc_session_id:3221500738, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337641865442, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337639866185, use_das=false, nested_level=0, session={this:0x7f92a22fdff8, id:3221500738, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3123ce0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:41.866107] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=13][errcode=-4012] fail to start stmt(ret=-4012, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:41.866113] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=6][errcode=-4012] fail start stmt(ret=-4012) [2024-09-26 16:00:41.866115] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4012] execute plan failed(ret=-4012) [2024-09-26 16:00:41.866117] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4012] open result set failed(ret=-4012) [2024-09-26 16:00:41.866118] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=1][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:41.866121] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:41.866124] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:41.866127] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:41.866128] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=1][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:41.866129] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-09-26 16:00:41.866131] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=1][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:41.866134] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1999949) [2024-09-26 16:00:41.866136] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885083-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:41.866140] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:41.866149] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:41.866151] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:41.866154] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8060, start=1727337639866181, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:41.866157] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:00:41.866160] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:41.866189] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:41.866206] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:41.866209] WDIAG [SHARE] get_snapshot_gc_scn (ob_global_stat_proxy.cpp:164) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:41.866212] WDIAG [STORAGE] get_global_info (ob_tenant_freeze_info_mgr.cpp:811) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to get global info(ret=-4012, tenant_id=1) [2024-09-26 16:00:41.866214] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:954) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] failed to get global info(ret=-4012) [2024-09-26 16:00:41.866216] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1008) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to try update info(tmp_ret=-4012, tmp_ret="OB_TIMEOUT") [2024-09-26 16:00:41.866222] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1044) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3] success to update min reserved snapshot(reserved_snapshot=0, duration=1800, snapshot_gc_ts_=0) [2024-09-26 16:00:41.866224] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1071) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2] update reserved snapshot finished(cost_ts=3, reserved_snapshot=0) [2024-09-26 16:00:41.878297] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24] success to notify_need_writing_throttling True [2024-09-26 16:00:41.878309] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.907004] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:41.907025] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337641907000) [2024-09-26 16:00:41.907031] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337641806238, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:41.907038] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:41.907049] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:41.907066] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:41.907392] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885752-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:41.907402] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885752-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:41.907407] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885752-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:41.907412] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885752-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:41.907416] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885752-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:41.907419] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885752-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:41.907423] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885752-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:41.907426] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885752-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:41.907450] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885752-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:41.907505] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337641907504, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=460, wlock_time=25, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:41.907517] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:41.907534] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223371143312, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:41.907540] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:41.907548] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:41.907554] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:41.907557] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:41.907563] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.907565] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:41.907568] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337641907521) [2024-09-26 16:00:41.908418] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:41.908428] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.915176] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337641815059]}) [2024-09-26 16:00:41.938474] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:41.938488] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.958454] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:41.968581] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:41.968593] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:41.968607] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:41.976678] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] tx gc loop thread is running(MTL_ID()=1) [2024-09-26 16:00:41.976689] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] try gc retain ctx [2024-09-26 16:00:41.977095] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-09-26 16:00:41.977124] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:41.999084] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:41.999092] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.002819] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337641002661]) [2024-09-26 16:00:42.002827] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337641002661], now0=[mts=1727337641002661], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:42.002833] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337641002443, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:42.002850] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:42.002857] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337641002443, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:42.002864] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337641002443, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:42.002881] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:42.002885] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:42.002887] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:42.002890] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:42.002892] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:42.002896] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=4, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:42.002900] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=4000, remain_us=24991432, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=4, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:42.006945] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:4, local_retry_times:4, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:42.006962] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:42.006965] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:42.006967] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:42.006974] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:42.007014] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:42.007017] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:42.007111] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:42.007122] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:42.007130] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337642007108) [2024-09-26 16:00:42.007146] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337641907034, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:42.007164] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223371242873, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:42.007171] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:42.007178] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:42.007184] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:42.007187] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:42.007193] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.007195] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.007197] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337642007149) [2024-09-26 16:00:42.009232] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500748, support_ssl=false) [2024-09-26 16:00:42.009265] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221500748, sess.client_addr_="127.0.0.1:36344") [2024-09-26 16:00:42.009281] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f9264004030, session_id:3221500748, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:42.009300] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:42.009314] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:42.009519] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9F-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:42.009717] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9F-0-0] [lt=7][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:42.009725] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9F-0-0] [lt=7][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"qT/+,<&Hw@yaYywi(+;+"}, ret=-4043) [2024-09-26 16:00:42.009732] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9F-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:42.009736] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9F-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:42.009744] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9F-0-0] [lt=3] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:42.009766] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9F-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500748, proxy_sessid:0}) [2024-09-26 16:00:42.009770] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9F-0-0] [lt=4] mark session id unused(sessid=3221500748) [2024-09-26 16:00:42.009777] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9F-0-0] [lt=5][errcode=0] server close connection(sessid=3221500748, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:42.009780] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9F-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500748, proxy_sessid=0) [2024-09-26 16:00:42.009782] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9F-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:42.009784] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384F9F-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500748, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:42.009818] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f9264004030, session_id:3221500748, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337642008760, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:42.009826] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] kill and revert session(conn.sessid_=3221500748, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:42.009829] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9264004030, session_id:3221500748, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337642008760, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:42.009834] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500748, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:42.015801] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337641815059]}) [2024-09-26 16:00:42.020634] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590C-0-0] [lt=51][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:42.020642] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590C-0-0] [lt=7][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:42.020649] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590C-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:42.020654] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590C-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:42.029491] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:42.029497] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.030883] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:42.030981] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=60, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:42.031604] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1384108, clean_num=31457) [2024-09-26 16:00:42.044615] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85883-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:42.044625] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85883-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:42.044651] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85883-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:42.050848] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:62, tid:3024}]) [2024-09-26 16:00:42.054857] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=40] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:42.055119] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:42.055126] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:42.055130] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:42.055133] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:42.055136] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:42.055138] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:42.055141] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:42.055143] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:42.055145] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:42.055147] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:42.055149] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:42.055151] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:42.055153] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:42.055155] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:42.055171] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:42.055174] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:42.055178] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:42.055180] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:42.055183] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:42.055186] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:42.055189] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:42.055194] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:42.055199] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:42.055201] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:42.055203] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:42.055208] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:42.055211] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:42.055214] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2579][RSAsyncTask0][T0][YB42C0A8002D-0006230057584E60-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:42.055217] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:42.055219] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:42.055221] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:42.055223] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337642055071, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:42.055226] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:42.055267] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:42.059263] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 10820.00) [2024-09-26 16:00:42.059272] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:42.059284] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:42.059288] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:42.059834] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:42.059840] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.084683] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:42.089937] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:42.089946] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.107162] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223371343540, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:42.107174] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:42.107183] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:42.107200] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:42.107204] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:42.107211] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.107213] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.107215] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337642107146) [2024-09-26 16:00:42.116347] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337641815059]}) [2024-09-26 16:00:42.116408] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D8B2-0-0] [lt=4][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337642116338}) [2024-09-26 16:00:42.120096] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:42.120102] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.150190] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:42.150197] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.158841] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=739216, replace_num=15728) [2024-09-26 16:00:42.158850] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:42.180441] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:42.180451] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.198512] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:00:42.198525] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=11] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:42.198536] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=5] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:42.207653] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:42.207669] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337642207650) [2024-09-26 16:00:42.207688] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337642007149, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:42.207707] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223371443187, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:42.207714] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:42.207722] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:42.207727] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:42.207730] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:42.207746] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.207749] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.207751] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337642207693) [2024-09-26 16:00:42.210485] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:42.210493] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.216613] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337641815059]}) [2024-09-26 16:00:42.231993] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:42.232120] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash time detail, (compute_wash_size_time=64, refresh_score_time=16, wash_time=1) [2024-09-26 16:00:42.232746] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1415565, clean_num=31457) [2024-09-26 16:00:42.240535] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:42.240542] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.242246] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:42.242261] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:42.242265] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:42.242268] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:42.242273] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:42.242275] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:42.242278] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:42.242281] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:42.242283] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:42.242284] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:42.242286] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:42.242289] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:42.242293] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:42.242296] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:42.242302] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:42.242304] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:42.242307] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:42.242309] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:42.242311] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:42.242314] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:42.242316] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:42.242322] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:42.242328] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:42.242331] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:42.242332] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:42.242338] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:42.242341] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:42.242344] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:42.242347] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:42.242349] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=1][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:42.242351] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:42.242353] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337642242174, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:42.242357] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:42.242359] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:42.242386] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:42.242389] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:42.242391] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:42.242393] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:42.242396] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:42.242398] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=1][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:42.242399] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485892-0-0] [lt=1][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:42.270922] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:42.270928] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.301080] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:42.301086] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.304669] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337642233354, now=[mts=1727337641303901]) [2024-09-26 16:00:42.304677] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337642233354, now=[mts=1727337641303901], now0=[mts=1727337641303901], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:42.304683] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269158080, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337641303265, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:42.304700] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269158080}) [2024-09-26 16:00:42.304709] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=9][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269158080, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337641303265, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:42.304731] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=21] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269158080, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337641303265, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337640303628, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269158080}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:42.304749] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:42.304754] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:42.304757] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:42.304763] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=5][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:42.304766] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:42.304769] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:42.304773] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:42.304780] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:42.304782] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:42.304783] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:42.304788] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:42.304791] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2001163) [2024-09-26 16:00:42.304794] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:42.304796] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:42.304799] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:42.304801] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=1][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:42.304804] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:42.304809] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:42.304834] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:42.304836] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:42.304839] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:42.304841] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:42.304845] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=4][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:42.304848] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:42.304955] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7B-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:42.304960] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:42.304962] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:42.304970] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=1] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:42.304977] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:42.305241] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:42.305247] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:42.308266] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:42.308278] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337642308263) [2024-09-26 16:00:42.308283] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337642207692, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:42.308317] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223371544777, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:42.308324] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:42.308349] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:42.308357] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:42.308361] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:42.308367] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.308369] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.308370] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337642308286) [2024-09-26 16:00:42.316112] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.316122] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337642316109], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:42.316139] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337642316109], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:42.316998] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337642316109]}) [2024-09-26 16:00:42.329744] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D3B5F-0-0] [lt=9][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:42.331626] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:42.331635] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.349386] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1727337642349385, bytes: 14639362, bw: 0.003365 MB/s, add_ts: 1003341, add_bytes: 3540 [2024-09-26 16:00:42.359004] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=4] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:42.362034] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:42.362044] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.386334] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:42.386335] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=24] current monitor number(seq_=-1) [2024-09-26 16:00:42.392209] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:42.392218] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.408805] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:42.408816] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337642408801) [2024-09-26 16:00:42.408822] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337642308286, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:42.408827] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:42.408836] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:42.408839] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:42.409182] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:42.409194] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:42.409199] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:42.409216] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:42.409220] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:42.409222] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:42.409226] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:42.409228] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:42.409230] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:42.409232] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:42.409234] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:42.409236] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:42.409239] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:42.409241] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:42.409248] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:42.409250] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:42.409253] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:42.409255] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:42.409257] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:42.409260] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:42.409262] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:42.409268] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:42.409273] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:42.409276] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:42.409277] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:42.409283] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:42.409286] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:42.409288] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885753-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:42.409291] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:42.409293] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:42.409295] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:42.409298] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337642409104, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:42.409301] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:42.409303] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:42.409342] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:42.409346] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337642409345, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=513, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:42.409352] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:42.409355] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:42.409370] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223371645370, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:42.409375] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:42.409380] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:42.409384] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:42.409386] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:42.409391] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.409393] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.409394] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337642409359) [2024-09-26 16:00:42.419587] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337642316109]}) [2024-09-26 16:00:42.422507] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:42.422516] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.433376] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:42.433503] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=50, refresh_score_time=14, wash_time=12) [2024-09-26 16:00:42.434108] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1447022, clean_num=31457) [2024-09-26 16:00:42.452765] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:42.452773] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.454733] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.454742] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.454742] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.454748] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.454748] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.454752] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.455694] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.455698] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.455700] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.455711] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.455719] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.455726] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=7][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.456664] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.456666] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.456671] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.456672] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.456674] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.456677] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.457654] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.457654] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.457660] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.457663] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.457663] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.457667] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.458660] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.458660] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.458668] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.458667] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.458671] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.458671] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.459581] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.459588] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.459594] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.459593] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.459601] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.459607] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.460127] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.460127] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.460132] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.460134] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.460136] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.460136] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.461549] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.461557] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.461558] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.461561] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.461562] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.461564] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:42.462513] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.462514] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.463471] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.463485] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.464490] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.464494] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.465467] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.465476] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.466407] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.466423] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.467432] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.467435] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.468032] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.468032] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.469355] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.469346] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.470330] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.470331] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=40][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.471304] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.471307] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.473264] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.473284] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.474245] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.474252] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.475205] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.475227] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=25][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.476206] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.476232] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.477161] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.477185] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.478164] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.478168] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.479110] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.479116] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.480086] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.480099] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.481085] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.481096] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.482070] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.482070] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.483012] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:42.483023] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.483020] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.483031] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.484017] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.484024] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.484548] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.484555] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.485953] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.485959] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.486513] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.486514] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.487921] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.487944] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.488472] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.488487] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.489844] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.489878] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.490847] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.490837] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.491803] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.491809] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.492769] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.492817] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.493770] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.493770] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.494298] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.494300] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.495711] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.495723] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.496270] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.496312] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.497678] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.497687] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.498666] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.498673] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.499609] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.499618] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.500613] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.500614] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.502559] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.502574] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.503517] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.503531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.504520] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.504531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:42.509450] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:42.509459] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337642509447) [2024-09-26 16:00:42.509478] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337642408825, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:42.509496] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223371744967, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:42.509502] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:42.509509] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:42.509514] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:42.509516] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:42.509522] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.509524] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.509526] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337642509481) [2024-09-26 16:00:42.513351] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:42.513357] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.520145] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337642316109]}) [2024-09-26 16:00:42.530031] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590D-0-0] [lt=31][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:42.530042] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590D-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:42.530051] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590D-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:42.530057] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590D-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:42.541633] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6440, mem_used=12478464) [2024-09-26 16:00:42.543594] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:42.543605] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.559225] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:42.563063] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:00:42.574105] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:42.574117] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.586878] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=6][errcode=-4018] try free send task failed(ret=-4018) [2024-09-26 16:00:42.602218] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.602238] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.602251] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.602258] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.604175] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:42.604208] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.609993] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:42.610010] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337642609989) [2024-09-26 16:00:42.610033] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337642509481, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:42.610065] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223371846519, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:42.610074] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:42.610085] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:42.610092] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:42.610091] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:42.610096] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:42.610104] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.610107] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.610109] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337642610039) [2024-09-26 16:00:42.621735] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337642316109]}) [2024-09-26 16:00:42.622283] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:42.622294] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:42.622299] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:42.622302] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:42.622305] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:42.622307] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:42.622311] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:42.622313] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:42.622315] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:42.622317] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:42.622319] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:42.622321] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:42.622323] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:42.622326] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:42.622332] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:42.622334] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:42.622336] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:42.622338] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:42.622356] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=17][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:42.622359] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:42.622362] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:42.622367] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:42.622373] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:42.622375] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:42.622376] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:42.622382] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:42.622386] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD3-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:42.622392] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2458][ServerGTimer][T0][YB42C0A8002D-0006230056A84DD3-0-0] [lt=5][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:42.622395] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:42.622398] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:42.622400] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:42.622402] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337642622218, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:42.622406] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:42.622448] WDIAG [SHARE] refresh (ob_all_server_tracer.cpp:568) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] fail to get servers_info(ret=-5019, ret="OB_TABLE_NOT_EXIST", GCTX.sql_proxy_=0x55c7486d2e88) [2024-09-26 16:00:42.622468] WDIAG [SHARE] runTimerTask (ob_all_server_tracer.cpp:626) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] fail to refresh all server map(ret=-5019) [2024-09-26 16:00:42.634447] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:42.634465] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.634609] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:42.634724] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=54, refresh_score_time=25, wash_time=2) [2024-09-26 16:00:42.634904] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855E5-0-0] [lt=9] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:42.635340] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1478479, clean_num=31457) [2024-09-26 16:00:42.659846] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:42.665758] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:42.665766] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.687186] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:262) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=12] ObArchiveSender is running(thread_index=0) [2024-09-26 16:00:42.687190] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:42.695934] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:42.695943] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.710637] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:42.710657] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337642710634) [2024-09-26 16:00:42.710662] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337642610039, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:42.710678] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223371947109, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:42.710686] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:42.710693] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:42.710707] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:42.710711] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:42.710716] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.710719] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.710721] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337642710665) [2024-09-26 16:00:42.722332] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337642316109]}) [2024-09-26 16:00:42.727201] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=26] success to notify_need_writing_throttling True [2024-09-26 16:00:42.727223] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.735023] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=6] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:42.757723] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:42.757733] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.760453] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:42.787834] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:42.787840] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.793618] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=14] PNIO [ratelimit] time: 1727337642793618, bytes: 0, bw: 0.000000 MB/s, add_ts: 1001112, add_bytes: 0 [2024-09-26 16:00:42.811211] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:42.811223] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337642811208) [2024-09-26 16:00:42.811229] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337642710665, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:42.811254] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223372046754, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:42.811261] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:42.811269] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:42.811274] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:42.811288] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:42.811297] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.811299] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.811301] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337642811231) [2024-09-26 16:00:42.817173] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:42.817181] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337642817170], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:42.817188] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337642817170], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:42.817191] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:42.817194] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:42.817953] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:42.817958] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.819987] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.820006] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.820015] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.820134] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.821946] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.821972] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.821979] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.821987] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.822009] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.822022] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.822029] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:42.822497] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337642817170]}) [2024-09-26 16:00:42.835749] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:42.835873] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=27, wash_time=2) [2024-09-26 16:00:42.836476] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1509936, clean_num=31457) [2024-09-26 16:00:42.849330] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:42.849337] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.880582] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:42.880588] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.910835] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:42.910843] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.911405] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:42.911417] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337642911402) [2024-09-26 16:00:42.911422] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337642811231, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:42.911427] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:42.911435] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:42.911439] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:42.911801] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:42.911811] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:42.911814] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:42.911817] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:42.911822] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:42.911824] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:42.911828] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:42.911830] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:42.911832] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:42.911834] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:42.911836] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:42.911838] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:42.911841] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:42.911843] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:42.911850] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:42.911853] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:42.911857] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:42.911861] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:42.911868] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=6][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:42.911873] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=4][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:42.911875] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:42.911881] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:42.911887] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:42.911889] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:42.911891] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:42.911896] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:42.911900] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:42.911902] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885754-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:42.911905] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:42.911907] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:42.911909] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:42.911911] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337642911713, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:42.911915] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:42.911918] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:42.911948] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:42.911953] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337642911953, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=521, wlock_time=9, check_leader_time=2, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:42.911960] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:42.911962] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:42.911979] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223372147397, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:42.911988] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:42.911994] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:42.911998] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:42.912000] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:42.912005] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.912008] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:42.912010] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337642911966) [2024-09-26 16:00:42.923115] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337642817170]}) [2024-09-26 16:00:42.928879] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C42-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2024-09-26 16:00:42.928891] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C42-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2024-09-26 16:00:42.928895] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C42-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2024-09-26 16:00:42.928899] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C42-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2024-09-26 16:00:42.928903] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C42-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:42.928905] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C42-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:42.928908] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C42-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2024-09-26 16:00:42.928910] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C42-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:42.928912] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C42-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:42.928914] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C42-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:42.928916] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C42-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:42.928935] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C42-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:42.942082] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:42.942105] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.961626] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:42.972446] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:42.972455] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:42.972487] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:42.977246] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:43.002648] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:43.002655] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.007530] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337642007013]) [2024-09-26 16:00:43.007538] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337642007013], now0=[mts=1727337642007013], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:43.007545] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337642006397, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:43.007572] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=26][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:43.007581] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337642006397, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:43.007590] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337642006397, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:43.007607] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:43.007612] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:43.007617] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:43.007620] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:43.007623] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:43.007627] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=5, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:43.007633] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] will sleep(sleep_us=5000, remain_us=23986700, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=5, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:43.010634] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500753, support_ssl=false) [2024-09-26 16:00:43.010678] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221500753, sess.client_addr_="127.0.0.1:36346") [2024-09-26 16:00:43.010686] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500753, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:43.010717] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:43.010728] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:43.010780] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9F-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:43.010959] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9F-0-0] [lt=6][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:43.010965] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9F-0-0] [lt=5][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"JniE]&AUj3]^@aI)0U;e"}, ret=-4043) [2024-09-26 16:00:43.010971] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9F-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:43.010978] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9F-0-0] [lt=6][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:43.010986] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9F-0-0] [lt=3] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:43.011008] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9F-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500753, proxy_sessid:0}) [2024-09-26 16:00:43.011012] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9F-0-0] [lt=4] mark session id unused(sessid=3221500753) [2024-09-26 16:00:43.011016] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9F-0-0] [lt=2][errcode=0] server close connection(sessid=3221500753, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:43.011019] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9F-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500753, proxy_sessid=0) [2024-09-26 16:00:43.011021] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9F-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:43.011024] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784F9F-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500753, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:43.011058] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500753, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337643010496, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:43.011067] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] kill and revert session(conn.sessid_=3221500753, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:43.011071] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500753, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337643010496, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:43.011075] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500753, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:43.011488] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:43.011496] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:43.011512] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337643011485) [2024-09-26 16:00:43.011517] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337642911424, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:43.011536] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223372246972, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:43.011542] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:43.011550] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:43.011556] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:43.011559] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:43.011565] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.011568] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.011570] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337643011520) [2024-09-26 16:00:43.013377] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:5, local_retry_times:5, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:43.013406] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=25][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:43.013410] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:43.013412] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:43.013417] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:43.013454] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:43.013458] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:43.023204] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=21] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337642817170]}) [2024-09-26 16:00:43.026237] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590E-0-0] [lt=59][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:43.026247] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590E-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:43.026255] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590E-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:43.026260] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590E-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:43.032961] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:43.032970] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.036953] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:43.037065] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=65, refresh_score_time=16, wash_time=1) [2024-09-26 16:00:43.037667] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1541393, clean_num=31457) [2024-09-26 16:00:43.045053] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85884-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:43.045065] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85884-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:43.045092] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85884-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:43.050515] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=37] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:43.055398] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:43.055696] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:43.055707] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:43.055711] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:43.055714] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:43.055717] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:43.055719] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:43.055724] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=4][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:43.055727] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:43.055729] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:43.055731] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:43.055733] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:43.055734] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:43.055736] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:43.055751] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E62-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:43.057356] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:89, tid:2579}]) [2024-09-26 16:00:43.063244] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:43.063252] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.063639] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 3216.00) [2024-09-26 16:00:43.063647] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:43.063660] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:43.063664] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:43.084714] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=7] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:43.093468] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:43.093475] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.112046] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:43.112057] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337643112043) [2024-09-26 16:00:43.112062] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337643011520, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:43.112079] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223372347610, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:43.112088] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:43.112107] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:43.112112] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:43.112115] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:43.112121] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.112125] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.112128] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337643112065) [2024-09-26 16:00:43.123758] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337642817170]}) [2024-09-26 16:00:43.123772] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D8BE-0-0] [lt=25][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337643123749}) [2024-09-26 16:00:43.123845] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:43.123854] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.154067] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:43.154076] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.161856] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:43.184323] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:43.184333] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.203366] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:43.203385] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=12] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:43.203391] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:43.212142] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:43.212159] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337643212138) [2024-09-26 16:00:43.212165] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337643112065, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:43.212182] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223372448495, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:43.212188] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:43.212196] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:43.212201] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:43.212204] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:43.212209] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.212212] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.212213] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337643212168) [2024-09-26 16:00:43.214441] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:43.214451] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.224308] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337642817170]}) [2024-09-26 16:00:43.238110] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:43.238212] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=53, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:43.238220] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1572850, clean_num=31457) [2024-09-26 16:00:43.242829] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:43.242850] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=19][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:43.242855] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:43.242859] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:43.242863] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:43.242865] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:43.242868] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:43.242871] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:43.242873] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:43.242875] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:43.242877] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:43.242879] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:43.242882] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:43.242885] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:43.242891] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:43.242893] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:43.242896] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:43.242898] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:43.242900] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:43.242903] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:43.242905] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:43.242914] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=7][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:43.242921] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:43.242923] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:43.242925] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:43.242931] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:43.242934] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:43.242937] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:43.242939] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:43.242941] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:43.242944] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:43.242946] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337643242758, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:43.242949] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:43.242952] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:43.242983] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=4][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:43.242988] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=4][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:43.242991] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=3][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:43.242993] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:43.242995] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=1][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:43.243000] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=5][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:43.243002] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485893-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:43.244560] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:43.244571] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.274698] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:43.274707] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.304763] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:43.304760] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337644234982, now=[mts=1727337642305240]) [2024-09-26 16:00:43.304769] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337644234982, now=[mts=1727337642305240], now0=[mts=1727337642305240], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:43.304769] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.304775] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269159e20, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337642304354, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:43.304795] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=19][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269159e20}) [2024-09-26 16:00:43.304802] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269159e20, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337642304354, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:43.304810] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269159e20, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337642304354, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337642305214, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269159e20}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:43.304829] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=18][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:43.304833] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:43.304836] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:43.304838] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:43.304841] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:43.304844] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:43.304847] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=1000135, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337644304982) [2024-09-26 16:00:43.304851] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:43.304857] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:43.304859] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:43.304860] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:43.304864] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:43.304897] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:43.304899] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:43.312205] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:43.312217] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337643312202) [2024-09-26 16:00:43.312223] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337643212168, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:43.312240] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223372548189, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:43.312248] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:43.312255] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:43.312260] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:43.312262] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:43.312267] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.312269] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.312273] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337643312226) [2024-09-26 16:00:43.316004] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD4-0-0] [lt=1][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-09-26 16:00:43.316011] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD4-0-0] [lt=7][errcode=-4002] refresh sever list failed(ret=-4002) [2024-09-26 16:00:43.316014] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD4-0-0] [lt=3][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-09-26 16:00:43.318054] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.318062] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337643318051], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:43.318069] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337643318051], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:43.324395] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337643318051]}) [2024-09-26 16:00:43.334351] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D3BA6-0-0] [lt=9][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:43.334824] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:43.334829] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.352814] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:218) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-09-26 16:00:43.352826] INFO [COMMON] handle (memory_dump.cpp:498) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-09-26 16:00:43.352852] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3] update_check_range(min_check_version=341, max_check_version=342, global_version=343) [2024-09-26 16:00:43.352943] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337643352942, bytes: 14643184, bw: 0.003632 MB/s, add_ts: 1003557, add_bytes: 3822 [2024-09-26 16:00:43.355353] INFO handle (memory_dump.cpp:626) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=6] statistics: tenant_cnt: 3, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 79 198 0 1 2 362 1 0 1 4 1 1 0 1 5 1 5 0 1 7 1 2 0 1 8 60 1 0 1 12 4 2 0 1 13 2 1 0 1 14 1 1 0 1 18 54 3 0 500 0 52 216 0 500 7 2 3 0 500 8 42 1 0 500 9 2 1 0 500 10 6 2 0 500 16 1 1 0 500 17 6 7 0 500 22 3 49 0 500 23 22 10 0 508 0 2 8 0 508 8 3 1 0 cost_time: 2504 [2024-09-26 16:00:43.361996] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:43.364945] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:43.364955] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.386399] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=27] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:43.386468] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=25] current monitor number(seq_=-1) [2024-09-26 16:00:43.387418] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [2645][Blacklist][T0][Y0-0000000000000000-0-0] [lt=11] blacklist_loop exec finished(cost_time=11, is_enabled=true, send_cnt=0) [2024-09-26 16:00:43.395273] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:43.395281] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.395924] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=1] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:43.395938] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:43.395944] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=5][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:43.395948] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:43.395953] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=5][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:43.395957] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=3] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:43.395959] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=1][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:43.395961] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:43.395977] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=14][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:43.395980] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:43.395982] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:43.395983] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:43.395988] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=5][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:43.395991] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:43.395994] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:43.395996] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:43.396002] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:43.396005] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:43.396008] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:43.396010] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:43.396015] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-09-26 16:00:43.396021] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:43.396026] WDIAG [SERVER] do_process (obmp_query.cpp:771) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=5][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-09-26 16:00:43.396032] WDIAG [SERVER] do_process (obmp_query.cpp:903) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, session={this:0x7f92a230bdf8, id:3221487638, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-09-26 16:00:43.396044] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2998][T1_L0_G0][T1][YB42C0A8002D-0006230059D84B29-0-0] [lt=8] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:43.407746] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] ====== tenant freeze timer task ====== [2024-09-26 16:00:43.407755] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:43.407785] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337640625515, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337640625479, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337640625532, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:43.407812] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:43.407913] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DA-0-0] [lt=2] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:43.407921] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DA-0-0] [lt=6] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:43.407926] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DA-0-0] [lt=4] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:43.407929] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DA-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:43.407933] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DA-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:43.407935] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DA-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:43.407937] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DA-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:43.407946] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DA-0-0] [lt=9] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:43.407948] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DA-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:43.407949] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DA-0-0] [lt=1] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:43.407952] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DA-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:43.407987] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851DA-0-0] [lt=3] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:43.412467] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:43.412478] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337643412465) [2024-09-26 16:00:43.412483] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337643312225, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:43.412488] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:43.412496] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:43.412498] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:43.412823] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:43.412833] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:43.412837] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:43.412840] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:43.412843] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:43.412845] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:43.412848] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:43.412851] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:43.412852] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:43.412854] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:43.412856] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:43.412858] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:43.412860] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:43.412862] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:43.412867] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:43.412869] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:43.412871] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:43.412873] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:43.412875] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:43.412877] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:43.412880] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:43.412884] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:43.412889] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:43.412891] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:43.412892] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:43.412897] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:43.412903] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=5][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:43.412905] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885755-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:43.412908] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:43.412911] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:43.412913] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:43.412915] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337643412746, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:43.412918] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:43.412921] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:43.412948] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:43.412950] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337643412950, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=458, wlock_time=8, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:43.412957] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:43.412962] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:43.412977] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223372648678, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:43.412982] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:43.412988] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:43.412992] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:43.412997] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:43.413002] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.413004] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.413005] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337643412966) [2024-09-26 16:00:43.424481] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337643318051]}) [2024-09-26 16:00:43.425319] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:43.425326] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.438404] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:43.438536] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=51, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:43.439120] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=0, clean_num=31457) [2024-09-26 16:00:43.455511] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:43.455523] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.455522] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.455530] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.455534] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.455570] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.455574] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.455578] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.456152] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.456157] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.456159] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.456182] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.456187] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.456189] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.456760] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.456765] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.456767] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.456793] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.456803] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.456807] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.457333] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.457338] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.457340] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.457366] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.457371] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.457373] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.457936] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.457941] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.457943] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.457944] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.457947] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.457949] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.458530] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.458533] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.458533] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.458535] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.458535] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.458537] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.459056] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.459060] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.459063] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.459077] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.459088] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.459096] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=8][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.459715] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.459725] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.459728] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.459733] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.459735] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.459737] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:43.460341] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.460373] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.460964] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.460966] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.461483] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.461519] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.462030] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.462068] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.462772] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.462774] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.463368] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.463382] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.463982] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.463982] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.464598] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.464602] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.465183] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.465214] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.465802] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.465806] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.466522] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.466644] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.467078] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.467222] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.467715] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.467820] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.468421] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.468421] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.469476] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.469477] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.470096] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.470102] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.470615] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.470620] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.471194] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.471237] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.471888] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.471901] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.472418] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.472439] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.473399] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.473415] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.474376] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.474380] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.474915] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.475015] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.475508] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.475711] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.476084] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.476338] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.476681] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.477015] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.477246] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.477554] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.478295] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.478306] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.478827] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.478910] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.479403] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.479493] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.480028] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.480048] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.480598] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.480659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.481216] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.481230] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.481848] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.481898] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.482515] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.482518] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.483165] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.483427] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.483707] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.484104] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.484304] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.484680] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.484819] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.485286] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.485455] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.485580] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:43.485585] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.485863] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.486062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.486484] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.486624] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.487076] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:43.490349] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8] LogLoopThread round_cost_time(us)(round_cost_time=8) [2024-09-26 16:00:43.513261] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:43.513273] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337643513257) [2024-09-26 16:00:43.513279] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337643412486, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:43.513296] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223372748937, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:43.513301] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:43.513309] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:43.513314] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:43.513316] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:43.513322] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.513324] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.513326] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337643513281) [2024-09-26 16:00:43.516053] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:43.516058] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.524004] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590F-0-0] [lt=44][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:43.524014] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590F-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:43.524023] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590F-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:43.524028] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718590F-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:43.524482] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337643318051]}) [2024-09-26 16:00:43.542421] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6450, mem_used=12478464) [2024-09-26 16:00:43.546204] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:43.546214] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.562736] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=15] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:43.563223] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=8] gc stale ls task succ [2024-09-26 16:00:43.576609] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:43.576615] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.603302] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.603305] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.603304] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.603324] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.606770] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:43.606781] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.613846] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:43.613859] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337643613843) [2024-09-26 16:00:43.613864] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337643513281, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:43.613882] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223372849538, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:43.613892] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:43.613899] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:43.613904] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:43.613907] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:43.613912] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.613915] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.613919] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337643613868) [2024-09-26 16:00:43.616228] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=8] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:43.616239] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:43.616243] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:43.616247] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:43.616250] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:43.616259] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:43.616263] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:43.616266] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:43.616267] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:43.616270] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:43.616272] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:43.616274] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:43.616276] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:43.616278] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:43.616285] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:43.616289] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:43.616291] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:43.616294] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:43.616296] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, ret=-5019) [2024-09-26 16:00:43.616298] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:43.616301] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:43.616306] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:43.616311] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:43.616313] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:43.616315] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:43.616320] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:43.616323] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:43.616325] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD4-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-09-26 16:00:43.616328] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:43.616330] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:43.616332] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:43.616334] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337643616160, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:43.616338] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:43.616340] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f92b4056738, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, columns_str="zone") [2024-09-26 16:00:43.616345] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:43.616372] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:634) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f92b4056738, where_condition="where svr_ip='192.168.0.45' and svr_port=2882", zone_name_holder=) [2024-09-26 16:00:43.616376] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:567) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:43.616378] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:576) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:43.616381] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:43.616386] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:43.625450] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337643318051]}) [2024-09-26 16:00:43.637016] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:43.637024] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.639364] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:43.639491] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=49, refresh_score_time=24, wash_time=2) [2024-09-26 16:00:43.640100] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=31457, clean_num=31457) [2024-09-26 16:00:43.659962] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:43.667141] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:43.667150] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.690470] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:43.697653] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:43.697661] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.713968] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:43.713984] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337643713966) [2024-09-26 16:00:43.713989] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337643613867, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:43.714005] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223372950137, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:43.714011] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:43.714020] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:43.714026] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:43.714028] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:43.714034] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.714036] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.714037] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337643713992) [2024-09-26 16:00:43.725472] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337643318051]}) [2024-09-26 16:00:43.728326] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:43.728351] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.758578] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:43.758587] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.762946] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:43.788704] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:43.788712] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.799619] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=41] PNIO [ratelimit] time: 1727337643799618, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006000, add_bytes: 0 [2024-09-26 16:00:43.814297] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:43.814307] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337643814294) [2024-09-26 16:00:43.814322] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337643713992, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:43.814340] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223373050758, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:43.814348] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:43.814357] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:43.814363] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:43.814366] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:43.814372] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.814374] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.814376] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337643814326) [2024-09-26 16:00:43.818393] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:43.818402] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337643818391], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:43.818408] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337643818391], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:43.818412] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:43.818416] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:43.818743] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:43.818748] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.821113] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.821116] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.821125] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.821128] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.821134] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.821156] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.821167] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.821176] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.821908] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.821910] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.821919] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:43.825753] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337643818391]}) [2024-09-26 16:00:43.840328] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:43.840431] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=49, refresh_score_time=16, wash_time=2) [2024-09-26 16:00:43.841071] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=62914, clean_num=31457) [2024-09-26 16:00:43.849256] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:43.849265] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.879422] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:43.879432] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.909516] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:43.909524] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.914608] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:43.914620] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337643914606) [2024-09-26 16:00:43.914626] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337643814325, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:43.914631] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:43.914639] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:43.914642] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:43.914973] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:43.914986] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:43.914990] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:43.914993] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:43.914996] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:43.915002] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=6][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:43.915006] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:43.915008] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:43.915010] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:43.915012] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:43.915014] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:43.915016] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:43.915019] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:43.915021] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:43.915028] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:43.915032] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:43.915035] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:43.915038] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:43.915040] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:43.915042] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:43.915045] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:43.915051] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:43.915058] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:43.915060] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:43.915062] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:43.915068] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:43.915071] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:43.915074] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885756-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:43.915076] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:43.915081] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:43.915083] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:43.915086] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337643914896, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:43.915089] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:43.915092] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:43.915121] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:43.915127] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337643915125, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=490, wlock_time=9, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:43.915134] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:43.915136] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:43.915151] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223373151381, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:43.915155] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:43.915161] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:43.915166] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:43.915168] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:43.915172] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.915174] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:43.915176] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337643915140) [2024-09-26 16:00:43.926430] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337643818391]}) [2024-09-26 16:00:43.939644] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:43.939655] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.939670] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] BlockGCTimerTask success(ret=0, cost_time_us=30, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.958566] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-09-26 16:00:43.958582] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=14][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-09-26 16:00:43.958587] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-09-26 16:00:43.958592] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-09-26 16:00:43.958596] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:43.958599] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:43.958602] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-09-26 16:00:43.958605] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:43.958607] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:43.958609] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:43.958611] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:43.958613] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:43.958616] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:43.958618] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:43.958643] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=21][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:43.958646] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:43.958649] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:43.958651] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:43.958653] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:43.958659] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=5][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:43.958662] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:43.958667] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:43.958673] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:43.958675] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:43.958677] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:43.958683] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:43.958686] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:43.958690] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD5-0-0] [lt=3][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:43.958721] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:109) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD5-0-0] [lt=0] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:43.958944] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-09-26 16:00:43.958962] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD5-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:43.958987] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:325) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD5-0-0] [lt=0] cal_all_part_disk_default_percentage succ(data_dir="/root/observer/store/sstable", clog_dir="/root/observer/store/clog", shared_mode=true, data_disk_total_size=99765694464, data_disk_default_percentage=60, clog_disk_total_size=99765694464, clog_disk_default_percentage=30) [2024-09-26 16:00:43.958995] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:361) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD5-0-0] [lt=8] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=99765694464, disk_size=15032385536) [2024-09-26 16:00:43.958998] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD5-0-0] [lt=3] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=99765694464) [2024-09-26 16:00:43.959002] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD5-0-0] [lt=3] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=99765694464, this={dir::"/root/observer/store/clog/log_pool", dir_fd:15, meta_fd:16, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:9388, max_block_id:9581, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2024-09-26 16:00:43.959007] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:133) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD5-0-0] [lt=5] refresh tenant config(tenants=[], ret=-5019) [2024-09-26 16:00:43.962983] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:43.969984] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5] success to notify_need_writing_throttling True [2024-09-26 16:00:43.969992] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:43.979386] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:44.000262] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:44.000271] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.000281] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:44.012263] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500758, support_ssl=false) [2024-09-26 16:00:44.012291] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221500758, sess.client_addr_="127.0.0.1:36348") [2024-09-26 16:00:44.012297] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500758, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:44.012315] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:44.012323] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2688][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:44.012370] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA0-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:44.012579] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA0-0-0] [lt=9][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:44.012586] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA0-0-0] [lt=6][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"HJzH8|.jM$h`*O/:vg~e"}, ret=-4043) [2024-09-26 16:00:44.012595] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA0-0-0] [lt=7][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:44.012599] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA0-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:44.012607] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA0-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:44.012627] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA0-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500758, proxy_sessid:0}) [2024-09-26 16:00:44.012634] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA0-0-0] [lt=6] mark session id unused(sessid=3221500758) [2024-09-26 16:00:44.012638] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA0-0-0] [lt=2][errcode=0] server close connection(sessid=3221500758, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:44.012640] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA0-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500758, proxy_sessid=0) [2024-09-26 16:00:44.012642] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA0-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:44.012645] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA0-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500758, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:44.012730] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500758, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337644011526, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:44.012741] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221500758, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:44.012744] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500758, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337644011526, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:44.012749] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500758, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:44.014717] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:44.014724] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:44.014730] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337644014702) [2024-09-26 16:00:44.014734] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337643914629, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:44.014753] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223373251010, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:44.014760] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:44.014778] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:44.014784] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:44.014786] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:44.014793] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.014795] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.014797] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337644014738) [2024-09-26 16:00:44.015329] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=21][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337643013453]) [2024-09-26 16:00:44.015341] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337643013453], now0=[mts=1727337643013453], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:44.015348] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337643011604, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:44.015364] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:44.015371] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337643011604, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:44.015379] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337643011604, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:44.015406] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=26][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:44.015418] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:44.015421] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:44.015424] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:44.015426] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:44.015431] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=6, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:44.015438] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5] will sleep(sleep_us=6000, remain_us=22978895, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=6, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:44.021496] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:6, local_retry_times:6, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:44.021507] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:44.021509] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:44.021510] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:44.021515] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:44.021558] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:44.021563] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:44.023612] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA0-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:44.023625] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA0-0-0] [lt=12][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:44.023683] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA0-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:44.023694] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:398) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=0] start to print blacklist info [2024-09-26 16:00:44.023714] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2] ls blacklist refresh finish(cost_time=407) [2024-09-26 16:00:44.025022] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185910-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:44.025031] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185910-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:44.025039] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185910-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:44.025044] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185910-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:44.026607] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337643818391]}) [2024-09-26 16:00:44.030527] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:44.030536] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.041197] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:44.041297] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:44.041937] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=94371, clean_num=31457) [2024-09-26 16:00:44.045593] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85885-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:44.045619] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85885-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:44.055888] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=6] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:44.056123] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E64-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:44.056142] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E64-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:44.061389] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:44.061399] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.066166] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 10779.00) [2024-09-26 16:00:44.066175] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:44.066187] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:44.066190] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:44.084900] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=8] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:44.091565] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:00:44.091575] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.114751] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:44.114763] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337644114749) [2024-09-26 16:00:44.114768] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337644014737, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:44.114797] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=223373350970, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:44.114806] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:44.114824] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:44.114829] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:44.114832] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:44.114838] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.114843] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.114845] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337644114783) [2024-09-26 16:00:44.121694] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:44.121699] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.126968] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337643818391]}) [2024-09-26 16:00:44.126989] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D8CA-0-0] [lt=11][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337644126961}) [2024-09-26 16:00:44.152043] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:44.152051] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.153000] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-09-26 16:00:44.153307] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD4-0-0] [lt=8] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-09-26 16:00:44.153318] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD4-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:44.153342] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD4-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:44.163136] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:44.182387] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:44.182395] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.198659] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] ====== check clog disk timer task ====== [2024-09-26 16:00:44.198666] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=6] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:44.198674] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:44.212526] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:44.212533] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.214872] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:44.214887] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337644214868) [2024-09-26 16:00:44.214892] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337644114783, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:44.214908] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223373451188, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:44.214916] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:44.214924] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:44.214928] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:44.214931] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:44.214936] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.214940] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.214943] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337644214895) [2024-09-26 16:00:44.227244] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337643818391]}) [2024-09-26 16:00:44.242178] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:44.242283] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash time detail, (compute_wash_size_time=55, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:44.242652] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:44.242660] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.242915] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=125828, clean_num=31457) [2024-09-26 16:00:44.243430] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:44.243444] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:44.243448] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:44.243455] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=6][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:44.243459] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:44.243461] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:44.243482] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:44.243486] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:44.243488] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:44.243490] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:44.243492] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:44.243494] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:44.243516] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485894-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:44.261598] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:217, tid:2919}]) [2024-09-26 16:00:44.272776] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:44.272785] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.303240] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:44.303250] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.306243] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337644234982, now=[mts=1727337643304896]) [2024-09-26 16:00:44.306253] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=9][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337644234982, now=[mts=1727337643304896], now0=[mts=1727337643304896], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:44.306259] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269159e20, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337643304523, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:44.306276] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269159e20}) [2024-09-26 16:00:44.306285] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269159e20, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337643304523, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:44.306293] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269159e20, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337643304523, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337642305214, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269159e20}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:44.306312] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=18][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:44.306317] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:44.306320] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:44.306322] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:44.306324] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:44.306328] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:44.306332] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:44.306338] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:44.306342] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:44.306344] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:44.306349] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:44.306363] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=12] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2001148) [2024-09-26 16:00:44.306366] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:44.306369] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=1][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:44.306373] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:44.306376] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:44.306378] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:44.306392] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:44.306420] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=4][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:44.306435] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=14][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:44.306438] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:44.306440] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=1][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:44.306443] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:44.306446] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:44.306573] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7C-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:44.306583] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:44.306586] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:44.306597] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:44.306601] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:44.306879] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:44.306885] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:44.315359] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:44.315378] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337644315355) [2024-09-26 16:00:44.315384] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337644214895, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:44.315404] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223373550873, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:44.315412] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:44.315428] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:44.315434] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:44.315437] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:44.315443] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.315445] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.315447] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337644315387) [2024-09-26 16:00:44.319325] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.319334] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337644319322], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:44.319340] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337644319322], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:44.319344] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337644319322]}) [2024-09-26 16:00:44.349649] WDIAG [STORAGE.TRANS] run1 (ob_clock_generator.cpp:113) [2625][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] clock out of order(cur_ts=1727337644349648, cur_ts_=1727337644321188, delta=28460) [2024-09-26 16:00:44.349669] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:44.349675] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.349698] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337644319322]}) [2024-09-26 16:00:44.360384] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D3C1F-0-0] [lt=8][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:44.360552] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337644360551, bytes: 14646724, bw: 0.003351 MB/s, add_ts: 1007609, add_bytes: 3540 [2024-09-26 16:00:44.363473] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=754944, replace_num=15728) [2024-09-26 16:00:44.363482] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:44.379749] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:44.379763] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.386577] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=10] current monitor number(seq_=-1) [2024-09-26 16:00:44.386645] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=9] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:44.410082] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:44.410096] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.415957] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:44.415969] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337644415953) [2024-09-26 16:00:44.415975] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337644315386, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:44.415981] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:44.415993] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:44.415996] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:44.416326] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:44.416335] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:44.416339] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:44.416343] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:44.416346] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:44.416349] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:44.416352] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:44.416354] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:44.416357] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:44.416358] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:44.416362] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=3][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:44.416364] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:44.416367] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:44.416368] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:44.416375] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:44.416378] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:44.416380] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:44.416382] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:44.416385] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:44.416388] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:44.416390] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:44.416397] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:44.416403] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:44.416405] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:44.416407] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:44.416412] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:44.416415] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:44.416418] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885757-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:44.416420] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:44.416423] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:44.416426] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:44.416428] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337644416248, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:44.416432] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:44.416435] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:44.416465] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:44.416469] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337644416468, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=480, wlock_time=10, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:44.416475] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:44.416478] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:44.416495] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223373652466, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:44.416500] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:44.416506] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:44.416510] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:44.416512] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:44.416518] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.416520] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.416523] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337644416481) [2024-09-26 16:00:44.440398] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24] success to notify_need_writing_throttling True [2024-09-26 16:00:44.440407] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.443391] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:44.443548] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=55, refresh_score_time=58, wash_time=2) [2024-09-26 16:00:44.444150] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=157285, clean_num=31457) [2024-09-26 16:00:44.450131] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337644319322]}) [2024-09-26 16:00:44.455962] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=19] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.455970] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.455975] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.456090] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.456134] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=43][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.456140] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.456503] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.456508] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.456512] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.456709] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.456717] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.456722] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.457040] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.457043] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.457045] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.457931] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.457940] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.457945] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.458001] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.458004] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.458006] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.458904] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.458912] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.458915] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.458917] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.458919] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.458924] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.459876] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.459875] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.459887] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.459890] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.459880] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.459896] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.460846] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.460852] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.460855] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.460868] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.460875] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.460882] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=7][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.461825] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.461830] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.461830] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.461833] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.461833] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.461835] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.462803] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.462803] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.462806] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:44.462808] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.463799] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.463799] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.464782] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.464800] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.465749] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.465779] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.466713] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.466724] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.467383] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.467391] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.468663] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.468670] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.469643] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.469653] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.470621] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:44.470628] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.470659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.470660] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.471598] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.471599] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.472582] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.472587] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.473543] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.473555] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.474520] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.474532] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.475501] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.475506] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.476484] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.476487] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.477220] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.477246] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.478464] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.478483] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.479421] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.479427] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.480383] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.480388] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.481363] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.481368] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.482348] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.482386] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.483333] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.483333] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.484307] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.484339] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.484861] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.484935] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.486247] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.486248] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.487240] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.487235] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.488200] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.488219] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=31][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.488787] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.488788] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.490146] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.490147] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.491123] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.491131] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.492097] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.492125] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.493117] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.493115] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.494076] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.494083] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.495071] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.495084] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.496011] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.496011] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.496990] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.496992] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.497976] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.498007] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.498947] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.498950] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.499927] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.499930] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.500895] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:44.500897] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.500901] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.500924] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.501896] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.501898] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.502845] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.502848] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.503839] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:44.516584] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:44.516596] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337644516581) [2024-09-26 16:00:44.516601] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337644415978, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:44.516617] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223373753048, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:44.516624] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:44.516642] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:44.516647] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:44.516649] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:44.516656] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.516658] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.516660] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337644516604) [2024-09-26 16:00:44.528494] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185911-0-0] [lt=34][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:44.528504] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185911-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:44.528513] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185911-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:44.528518] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185911-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:44.531237] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:44.531244] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.542941] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6462, mem_used=12478464) [2024-09-26 16:00:44.550744] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337644319322]}) [2024-09-26 16:00:44.561500] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:44.561507] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.563435] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=6] gc stale ls task succ [2024-09-26 16:00:44.563583] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:44.591814] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:44.591822] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.601502] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.601506] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.601517] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.601603] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.617136] WDIAG [PALF] convert_to_ts (scn.cpp:265) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2024-09-26 16:00:44.617144] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1727337644617135, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1727337644617135, force_self_check=false, weak_read_refresh_interval=100000) [2024-09-26 16:00:44.617159] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:44.617165] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337644617157) [2024-09-26 16:00:44.617169] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337644516604, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:44.617196] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=223373853598, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:44.617202] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:44.617209] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:44.617213] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:44.617216] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:44.617222] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.617224] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.617226] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337644617182) [2024-09-26 16:00:44.617538] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:44.623015] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:44.623022] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.644646] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:44.644760] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:44.645361] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=188742, clean_num=31457) [2024-09-26 16:00:44.651364] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337644319322]}) [2024-09-26 16:00:44.653311] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:44.653319] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.660111] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:44.683593] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:44.683602] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.692379] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:44.713945] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:44.713955] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.717759] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:44.717774] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337644717755) [2024-09-26 16:00:44.717780] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337644617182, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:44.717792] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get_max_applied_scn(scn={val:1727114270763524000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:374, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:1727114270763524000, v:0}, max_applied_cb_scn_:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:44.717801] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] get_min_unreplayed_log_info(lsn={lsn:629872191939}, scn={val:1727114270763524001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:00:44.717820] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=223373953297, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:44.717825] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:44.717832] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:44.717836] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:44.717838] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:44.717844] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.717846] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.717848] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337644717783) [2024-09-26 16:00:44.745094] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:44.745121] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.752002] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337644319322]}) [2024-09-26 16:00:44.758772] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=3] XA scheduler heartbeat task statistics(avg_time=0) [2024-09-26 16:00:44.764624] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:44.775423] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:44.775432] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.781212] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=4] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-09-26 16:00:44.781221] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=6] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-09-26 16:00:44.781227] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=5] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=689) [2024-09-26 16:00:44.799783] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337644799782, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000164, add_bytes: 0 [2024-09-26 16:00:44.805746] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:44.805755] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.818381] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:44.818391] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337644818378) [2024-09-26 16:00:44.818396] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337644717782, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:44.818433] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223374053875, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:44.818440] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:44.818447] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:44.818452] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:44.818469] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:44.818476] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.818478] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.818480] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337644818399) [2024-09-26 16:00:44.820318] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.820319] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.820329] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.820334] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.820341] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.820334] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.820431] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:44.820440] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337644820428], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:44.820461] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337644820428], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:44.820464] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:44.820468] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:44.821255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.821267] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.821419] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.821429] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.821746] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:44.835919] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:44.835927] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.845785] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:44.845891] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=54, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:44.846621] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=220199, clean_num=31457) [2024-09-26 16:00:44.852541] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337644820428]}) [2024-09-26 16:00:44.866242] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:44.866249] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.866611] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, table_name.ptr()="data_size:16, data:5F5F616C6C5F6D657267655F696E666F", ret=-5019) [2024-09-26 16:00:44.866623] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, ret=-5019) [2024-09-26 16:00:44.866628] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_merge_info, db_name=oceanbase) [2024-09-26 16:00:44.866631] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_merge_info) [2024-09-26 16:00:44.866634] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:44.866637] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:44.866640] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_merge_info' doesn't exist [2024-09-26 16:00:44.866643] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:44.866645] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:44.866647] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:44.866652] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:44.866654] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:44.866657] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:44.866659] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:44.866666] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:44.866668] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:44.866672] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:44.866675] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:44.866679] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_merge_info WHERE tenant_id = '1', ret=-5019) [2024-09-26 16:00:44.866682] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:44.866685] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:44.866691] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:44.866696] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:44.866699] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:44.866700] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:44.866706] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:44.866710] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:44.866713] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885084-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, aret=-5019, ret=-5019) [2024-09-26 16:00:44.866715] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:44.866718] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:44.866721] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:44.866725] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337644866545, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:44.866729] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:44.866732] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:49) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:44.866762] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:890) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] failed to load global merge info(ret=-5019, ret="OB_TABLE_NOT_EXIST", global_merge_info={tenant_id:1, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2024-09-26 16:00:44.866775] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1005) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] fail to refresh merge info(tmp_ret=-5019, tmp_ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:44.866990] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:44.866994] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:44.896467] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:44.896475] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.917960] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] ====== checkpoint timer task ====== [2024-09-26 16:00:44.918419] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] succ to get rec scn(*this={this:0x7f92a6a04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1726077600526835000, v:0}, last_push_gc_task_ts_:1727334198745636, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:44.918425] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:44.918435] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337644918422) [2024-09-26 16:00:44.918431] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] tx ctx memtable get rec scn(this={ObITable:{this:0x7f92a22a4080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f92a22a4080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:44.918443] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337644818399, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:44.918444] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1723174350926847000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-09-26 16:00:44.918449] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:44.918476] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:44.918479] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:44.918473] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:44.918497] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}, min_rec_scn={val:1726163586617842000, v:0}, ls_id_={id:1}) [2024-09-26 16:00:44.918512] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:44.918516] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=4] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:44.918518] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=1] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:44.918524] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=1] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1726163586617842000, v:0}, checkpoint_scn_in_ls_meta={val:1726163586617842000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-09-26 16:00:44.918534] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}, curr_base_lsn={lsn:627966418944}, new_base_lsn={lsn:627966418944}, lsn={lsn:627966418944}) [2024-09-26 16:00:44.918538] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=4] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:627966418944}, ls->get_ls_id()={id:1}) [2024-09-26 16:00:44.918542] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-09-26 16:00:44.918789] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:44.918799] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:44.918803] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:44.918806] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:44.918809] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:44.918813] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:44.918817] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:44.918819] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:44.918821] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:44.918823] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:44.918825] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:44.918827] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:44.918829] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:44.918831] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:44.918839] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:44.918841] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:44.918844] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:44.918846] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:44.918848] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:44.918850] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:44.918852] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:44.918859] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=6][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:44.918876] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:44.918879] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:44.918883] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=3][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:44.918889] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:44.918892] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:44.918894] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885758-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:44.918897] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:44.918901] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:44.918904] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:44.918906] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337644918723, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:44.918909] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:44.918912] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:44.918938] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:44.918941] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337644918941, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=475, wlock_time=18, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:44.918948] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:44.918951] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:44.918965] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223374154470, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:44.918969] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:44.918976] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:44.918982] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:44.918985] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:44.918989] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.918991] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:44.918993] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337644918956) [2024-09-26 16:00:44.927777] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:44.927784] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.952720] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337644820428]}) [2024-09-26 16:00:44.957954] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:44.957964] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:44.964688] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:44.978596] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:44.981795] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-09-26 16:00:44.981803] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=6] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:44.981822] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=18] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:44.981828] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=5] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:44.983491] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] Dumping All Cache Objs(alloc_obj_list.count()=3, alloc_obj_list=[{obj_id:78, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157779}, {obj_id:79, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:5, added_to_lc:true, mem_used:175172}]) [2024-09-26 16:00:44.983538] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=46] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:44.989284] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=91] success to notify_need_writing_throttling True [2024-09-26 16:00:44.989299] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.013877] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500765, support_ssl=false) [2024-09-26 16:00:45.013903] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221500765, sess.client_addr_="127.0.0.1:36350") [2024-09-26 16:00:45.013910] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f9264004030, session_id:3221500765, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:45.013937] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:45.013957] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:45.013963] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:280) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=5] mysql login queue(mysql_queue->queue_.size()=1) [2024-09-26 16:00:45.014010] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA0-0-0] [lt=5] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:45.014247] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA0-0-0] [lt=6][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:45.014255] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA0-0-0] [lt=7][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"N"Z!2x'Xdmw0%fWaaAbH"}, ret=-4043) [2024-09-26 16:00:45.014261] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA0-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:45.014265] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA0-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:45.014273] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA0-0-0] [lt=3] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:45.014298] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA0-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500765, proxy_sessid:0}) [2024-09-26 16:00:45.014302] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA0-0-0] [lt=3] mark session id unused(sessid=3221500765) [2024-09-26 16:00:45.014308] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA0-0-0] [lt=5][errcode=0] server close connection(sessid=3221500765, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:45.014311] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA0-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500765, proxy_sessid=0) [2024-09-26 16:00:45.014314] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA0-0-0] [lt=3][errcode=-4016] session is null [2024-09-26 16:00:45.014316] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA0-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500765, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:45.014345] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f9264004030, session_id:3221500765, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337645013866, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:45.014387] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=40] kill and revert session(conn.sessid_=3221500765, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:45.014391] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9264004030, session_id:3221500765, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337645013866, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:45.014395] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500765, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:45.018561] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:45.018569] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:45.018590] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337645018557) [2024-09-26 16:00:45.018595] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337644918447, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:45.018625] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=223374254098, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.018634] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:45.018642] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:45.018647] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:45.018650] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:45.018655] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.018657] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.018659] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337645018610) [2024-09-26 16:00:45.019531] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:45.019541] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.019559] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:45.021505] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337644021557]) [2024-09-26 16:00:45.021541] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=36][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337644021557], now0=[mts=1727337644021557], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:45.021548] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337644021418, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:45.021564] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:45.021572] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337644021418, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:45.021579] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337644021418, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:45.021596] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:45.021601] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:45.021604] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:45.021608] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:45.021611] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:45.021615] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=7, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:45.021630] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13] will sleep(sleep_us=7000, remain_us=21972703, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=7, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:45.029369] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:7, local_retry_times:7, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:45.029384] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:45.029445] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=60][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:45.029449] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:45.029477] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=25][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:45.029518] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:45.029521] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:45.029531] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3538][T1_L0_G2][T1][YB42C0A8002D-0006230057185912-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:45.029539] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3538][T1_L0_G2][T1][YB42C0A8002D-0006230057185912-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:45.029547] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3538][T1_L0_G2][T1][YB42C0A8002D-0006230057185912-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:45.029552] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3538][T1_L0_G2][T1][YB42C0A8002D-0006230057185912-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:45.047225] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:45.047238] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:45.047243] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:45.047258] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:45.047261] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:45.047263] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:45.047276] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:45.047281] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:45.047283] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:45.047284] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:45.047286] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:45.047292] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:45.047294] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:45.047297] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:45.047304] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:45.047306] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:45.047309] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:45.047310] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:45.047313] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:45.047315] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:45.047317] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:45.047321] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:45.047325] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:45.047327] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:45.047328] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:45.047334] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:45.047337] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:45.047339] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85886-0-0] [lt=1][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:45.047341] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:45.047344] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:45.047345] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85886-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:45.047348] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337645047159, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:45.047351] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85886-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:45.047353] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85886-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:45.047374] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:45.047376] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85886-0-0] [lt=2][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:45.047552] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:45.047651] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash time detail, (compute_wash_size_time=65, refresh_score_time=16, wash_time=2) [2024-09-26 16:00:45.048338] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=251656, clean_num=31457) [2024-09-26 16:00:45.049821] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.049829] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.052774] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337644820428]}) [2024-09-26 16:00:45.057653] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=6] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:45.057939] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:45.057950] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:45.057969] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=19][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:45.057973] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:45.057977] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:45.057979] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:45.057982] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:45.057984] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:45.057986] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:45.057987] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:45.057989] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:45.057993] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:45.057996] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:45.057998] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:45.058004] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:45.058006] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:45.058008] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:45.058011] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:45.058013] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:45.058015] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:45.058017] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:45.058024] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=6][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:45.058029] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:45.058031] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:45.058032] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:45.058037] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:45.058040] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E66-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:45.058044] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2579][RSAsyncTask0][T0][YB42C0A8002D-0006230057584E66-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:45.058048] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:45.058051] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:45.058053] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:45.058055] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337645057891, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:45.058059] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:45.058104] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:45.070382] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 12169.00) [2024-09-26 16:00:45.070391] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:45.070402] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:45.070405] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:45.080068] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:45.080078] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.084954] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:45.110400] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:45.110384] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=15] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:45.110420] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.119155] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:45.119168] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337645119152) [2024-09-26 16:00:45.119173] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337645018610, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:45.119200] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223374354727, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.119208] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:45.119216] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:45.119221] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:45.119224] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:45.119229] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.119232] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.119233] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337645119176) [2024-09-26 16:00:45.140655] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:45.140685] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.153359] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337644820428]}) [2024-09-26 16:00:45.153372] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D8D6-0-0] [lt=10][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337645153354}) [2024-09-26 16:00:45.165070] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:45.170936] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.170944] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.185244] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337644820428]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:45.185259] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:45.201291] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.201301] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.204491] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:45.204505] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:45.204510] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:45.219512] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:45.219529] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337645219508) [2024-09-26 16:00:45.219535] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337645119176, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:45.219551] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223374455106, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.219557] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:45.219564] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:45.219569] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:45.219571] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:45.219577] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.219578] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.219581] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337645219538) [2024-09-26 16:00:45.231904] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:45.231914] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.243967] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:45.243984] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=16][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:45.243990] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:45.243993] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:45.243996] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:45.243999] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:45.244002] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:45.244005] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:45.244007] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:45.244038] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=30][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:45.244040] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:45.244042] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:45.244045] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:45.244047] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:45.244063] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:45.244066] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:45.244069] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:45.244072] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:45.244077] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:45.244081] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:45.244084] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:45.244090] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:45.244096] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:45.244099] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:45.244100] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:45.244107] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:45.244117] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=9][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:45.244121] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=4][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:45.244124] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485895-0-0] [lt=3][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.248556] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:45.248672] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=53, refresh_score_time=26, wash_time=2) [2024-09-26 16:00:45.249315] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=283113, clean_num=31457) [2024-09-26 16:00:45.253392] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337644820428]}) [2024-09-26 16:00:45.262745] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.262757] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.282736] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=11][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337644820428]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:45.282753] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=16][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:45.292813] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:45.292822] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.307661] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=18][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337646236605, now=[mts=1727337644306877]) [2024-09-26 16:00:45.307669] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337646236605, now=[mts=1727337644306877], now0=[mts=1727337644306877], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:45.307675] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926915bbc0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337644306616, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:45.307704] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=28][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915bbc0}) [2024-09-26 16:00:45.307713] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926915bbc0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337644306616, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:45.307721] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926915bbc0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337644306616, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337644306849, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915bbc0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:45.307738] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:45.307743] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:45.307749] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:45.307752] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:45.307755] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:45.307759] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:45.307764] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=998841, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337646306605) [2024-09-26 16:00:45.307768] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:45.307775] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:45.307777] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:45.307778] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:45.307784] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:45.307821] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:45.307826] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:45.319545] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:45.319557] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337645319542) [2024-09-26 16:00:45.319563] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337645219538, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:45.319581] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223374555754, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.319587] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:45.319594] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:45.319599] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:45.319601] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:45.319607] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.319609] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.319611] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337645319566) [2024-09-26 16:00:45.320847] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.320858] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337645320844], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:45.320865] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337645320844], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:45.322858] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:45.322868] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.353097] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:45.353107] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.353669] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337645320844]}) [2024-09-26 16:00:45.364855] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D3C7E-0-0] [lt=15][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:45.365157] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:45.368254] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337645368253, bytes: 14650546, bw: 0.003617 MB/s, add_ts: 1007702, add_bytes: 3822 [2024-09-26 16:00:45.383280] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:45.383290] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.386683] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=10] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:45.387491] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=11] current monitor number(seq_=-1) [2024-09-26 16:00:45.413472] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.413491] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.414600] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] ====== tenant freeze timer task ====== [2024-09-26 16:00:45.414610] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:45.414640] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337640625515, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337640625479, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337640625532, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:45.414665] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:45.414742] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DB-0-0] [lt=2] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:45.414754] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DB-0-0] [lt=9] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:45.414760] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DB-0-0] [lt=4] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:45.414765] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DB-0-0] [lt=4] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:45.414769] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DB-0-0] [lt=4][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:45.414772] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DB-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:45.414774] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DB-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:45.414784] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DB-0-0] [lt=9] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:45.414787] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DB-0-0] [lt=3][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:45.414789] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DB-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:45.414793] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DB-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:45.414837] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851DB-0-0] [lt=9] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:45.419953] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:45.419966] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337645419949) [2024-09-26 16:00:45.419974] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337645319566, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:45.419981] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:45.420005] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:45.420010] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:45.420429] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885759-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:45.420439] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885759-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.420479] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885759-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:45.420548] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337645420547, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=546, wlock_time=11, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:45.420560] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:45.420581] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] get wrs ts(ls_id={id:1}, delta=223374655575, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.420588] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:45.420596] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:45.420602] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:45.420605] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:45.420612] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.420615] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.420618] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337645420566) [2024-09-26 16:00:45.443669] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:45.443686] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.449676] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:45.449791] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=53, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:45.450402] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=314570, clean_num=31457) [2024-09-26 16:00:45.453760] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337645320844]}) [2024-09-26 16:00:45.457103] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.457113] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.457119] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.457136] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.457142] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.457156] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=13][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.457166] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.457170] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.457197] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=27][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.458071] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.458080] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.458084] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.458080] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.458088] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.458092] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.458097] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.458099] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.458101] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.458634] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.458642] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.458645] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.458690] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.458694] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.458697] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.459002] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.459009] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.459013] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.459181] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.459186] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.459188] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.459236] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.459239] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.459241] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.459674] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.459683] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.459687] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.459894] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.459899] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.459902] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.459905] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.459906] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.459907] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.460227] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.460231] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.460234] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.460459] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.460462] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.460464] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:45.460467] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.460801] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.461065] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.461072] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.461345] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.461954] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.461963] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.462018] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.462651] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.462657] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.462663] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.463270] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.463307] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.463312] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.463817] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.463845] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.463850] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.464422] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.464432] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.464436] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.465025] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.465033] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.465061] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.465664] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.465670] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.465714] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.466336] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.466338] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.466343] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.466934] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.466940] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.466990] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.467450] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.467466] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.467551] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.467582] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:47, tid:2932}]) [2024-09-26 16:00:45.468105] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.468136] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.468141] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.468772] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.468773] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.468778] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.469411] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.469414] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.469418] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.470011] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.470013] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.470018] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.470537] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.470643] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.470649] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.471056] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.471279] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.471284] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.471633] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.471856] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.471863] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.472159] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.472460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.472465] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.472796] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.473058] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.473064] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.473447] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.473646] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.473651] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.473775] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=34] success to notify_need_writing_throttling True [2024-09-26 16:00:45.473780] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.474093] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.474225] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.474229] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.474666] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.474803] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.474808] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.475236] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.475383] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.475387] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.475811] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.475952] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.475962] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.476331] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.476487] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.476498] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.477042] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.477282] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.477290] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.477640] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:45.503948] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:45.503956] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.520021] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:45.520030] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337645520018) [2024-09-26 16:00:45.520035] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337645419978, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:45.520051] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223374755464, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.520056] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:45.520063] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:45.520067] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:45.520069] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:45.520074] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.520076] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.520078] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337645520038) [2024-09-26 16:00:45.526666] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185913-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:45.526677] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185913-0-0] [lt=11][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:45.526686] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185913-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:45.526693] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185913-0-0] [lt=6][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:45.534211] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.534221] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.543017] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6470, mem_used=12478464) [2024-09-26 16:00:45.553797] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337645320844]}) [2024-09-26 16:00:45.563517] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:00:45.564519] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.564530] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.565253] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:45.589948] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [2619][DumpLSLoc][T0][YB42C0A8002D-0006230058384C7A-0-0] [lt=15] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1727337640934007, replica_locations:[{server:"192.168.0.45:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:374}]}]) [2024-09-26 16:00:45.591815] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=1] TIMER THREAD STAT: (thread_id=2513, task_cnt=298, avg_time=847) [2024-09-26 16:00:45.591832] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=16] TIMER THREAD STAT: (thread_id=2514, task_cnt=300, avg_time=42) [2024-09-26 16:00:45.591835] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=3] TIMER THREAD STAT: (thread_id=2817, task_cnt=1973, avg_time=30) [2024-09-26 16:00:45.591837] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2899, task_cnt=2931, avg_time=0) [2024-09-26 16:00:45.591839] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2820, task_cnt=120, avg_time=3) [2024-09-26 16:00:45.591841] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2458, task_cnt=168, avg_time=1068) [2024-09-26 16:00:45.591843] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2643, task_cnt=60, avg_time=1) [2024-09-26 16:00:45.591845] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=1] TIMER THREAD STAT: (thread_id=2632, task_cnt=60, avg_time=11) [2024-09-26 16:00:45.591847] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2898, task_cnt=61, avg_time=3) [2024-09-26 16:00:45.591850] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=3] TIMER THREAD STAT: (thread_id=2838, task_cnt=60, avg_time=1) [2024-09-26 16:00:45.591852] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2919, task_cnt=60, avg_time=492) [2024-09-26 16:00:45.591863] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=11] TIMER THREAD STAT: (thread_id=2952, task_cnt=248, avg_time=1) [2024-09-26 16:00:45.591865] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2460, task_cnt=20, avg_time=7) [2024-09-26 16:00:45.591867] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2692, task_cnt=20, avg_time=10) [2024-09-26 16:00:45.591869] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2935, task_cnt=30, avg_time=26) [2024-09-26 16:00:45.591871] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2892, task_cnt=18, avg_time=1334531) [2024-09-26 16:00:45.591873] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2461, task_cnt=12, avg_time=20) [2024-09-26 16:00:45.591875] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2823, task_cnt=12, avg_time=1) [2024-09-26 16:00:45.591877] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=1] TIMER THREAD STAT: (thread_id=2837, task_cnt=12, avg_time=1713) [2024-09-26 16:00:45.591878] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=1] TIMER THREAD STAT: (thread_id=2894, task_cnt=12, avg_time=448) [2024-09-26 16:00:45.591881] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2933, task_cnt=12, avg_time=560) [2024-09-26 16:00:45.591884] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2934, task_cnt=12, avg_time=1290) [2024-09-26 16:00:45.591886] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=1] TIMER THREAD STAT: (thread_id=2937, task_cnt=12, avg_time=23) [2024-09-26 16:00:45.591887] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=1] TIMER THREAD STAT: (thread_id=2936, task_cnt=12, avg_time=26) [2024-09-26 16:00:45.591889] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2463, task_cnt=6, avg_time=516) [2024-09-26 16:00:45.591891] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2907, task_cnt=6, avg_time=55) [2024-09-26 16:00:45.591893] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2571, task_cnt=62, avg_time=16) [2024-09-26 16:00:45.591895] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2545, task_cnt=30, avg_time=0) [2024-09-26 16:00:45.591897] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2464, task_cnt=60, avg_time=467) [2024-09-26 16:00:45.591899] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=3684, task_cnt=60, avg_time=9) [2024-09-26 16:00:45.591901] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=3691, task_cnt=60, avg_time=771) [2024-09-26 16:00:45.591903] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=1] TIMER THREAD STAT: (thread_id=2618, task_cnt=59, avg_time=281) [2024-09-26 16:00:45.591904] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=1] TIMER THREAD STAT: (thread_id=3026, task_cnt=30, avg_time=295) [2024-09-26 16:00:45.591906] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2617, task_cnt=12, avg_time=20) [2024-09-26 16:00:45.591908] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2635, task_cnt=6, avg_time=16) [2024-09-26 16:00:45.591910] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2644, task_cnt=6, avg_time=11) [2024-09-26 16:00:45.591912] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2619, task_cnt=6, avg_time=34) [2024-09-26 16:00:45.591914] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2947, task_cnt=2, avg_time=15) [2024-09-26 16:00:45.591916] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2638, task_cnt=1, avg_time=2) [2024-09-26 16:00:45.591918] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2944, task_cnt=0, avg_time=0) [2024-09-26 16:00:45.591920] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2633, task_cnt=0, avg_time=0) [2024-09-26 16:00:45.591922] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=2] TIMER THREAD STAT: (thread_id=2920, task_cnt=0, avg_time=0) [2024-09-26 16:00:45.591924] INFO dump (ob_timer_monitor.cpp:200) [2644][ObTimer][T0][Y0-0000000000000000-0-0] [lt=1] TIMER THREAD STAT: (thread_id=3685, task_cnt=0, avg_time=0) [2024-09-26 16:00:45.594778] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:45.594789] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.601479] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [2635][LocalityReload][T0][Y0-0000000000000000-0-0] [lt=34] runTimer to refresh locality_info(ret=0) [2024-09-26 16:00:45.601500] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.601966] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:45.601985] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=18][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:45.601990] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:45.601993] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:45.601997] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:45.601999] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:45.602003] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:45.602005] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:45.602007] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:45.602009] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:45.602011] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] resolve table failed(ret=-5019) [2024-09-26 16:00:45.602013] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:2822) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-09-26 16:00:45.602015] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2676) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] resolve joined table failed(ret=-5019) [2024-09-26 16:00:45.602017] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:45.602019] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:45.602022] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:45.602024] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:45.602030] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:45.602033] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:45.602036] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:45.602038] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:45.602041] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name, ret=-5019) [2024-09-26 16:00:45.602044] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:45.602046] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:45.602052] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:45.602057] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:45.602059] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:45.602060] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:45.602067] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:45.602070] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7A-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:45.602073] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2634][LocltyRefTask][T0][YB42C0A8002D-0006230058284C7A-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name"}, aret=-5019, ret=-5019) [2024-09-26 16:00:45.602076] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name) [2024-09-26 16:00:45.602079] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:45.602080] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:45.602083] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337645601862, sql=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name) [2024-09-26 16:00:45.602086] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:45.602089] WDIAG [SHARE] load_region (ob_locality_table_operator.cpp:158) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, sql=select svr_ip, svr_port, a.zone, info, value, b.name, a.status, a.start_service_time, a.stop_time from __all_server a LEFT JOIN __all_zone b ON a.zone = b.zone WHERE (b.name = 'region' or b.name = 'idc' or b.name = 'status' or b.name = 'zone_type') and a.zone != '' order by svr_ip, svr_port, b.name) [2024-09-26 16:00:45.602091] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2] load region(ret=-5019, locality_info={version:0, local_region:"", local_zone:"", local_idc:"", local_zone_type:3, local_zone_status:3, locality_region_array:[], locality_zone_array:[]}) [2024-09-26 16:00:45.602121] WDIAG [STORAGE] load_region (ob_locality_manager.cpp:236) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] localitity operator load region error(ret=-5019) [2024-09-26 16:00:45.602125] WDIAG [STORAGE] process (ob_locality_manager.cpp:762) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] process refresh locality task fail(ret=-5019) [2024-09-26 16:00:45.602463] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.602474] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.603588] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.620057] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:45.620207] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:45.620223] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337645620203) [2024-09-26 16:00:45.620230] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337645520038, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:45.620250] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223374855747, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.620256] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:45.620265] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:45.620270] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:45.620273] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:45.620280] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.620283] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.620284] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337645620233) [2024-09-26 16:00:45.625094] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:00:45.625102] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.627563] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] ====== traversal_flush timer task ====== [2024-09-26 16:00:45.627590] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272926030, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6163825430799493735, dag_cnt=1, dag_type_cnts=1) [2024-09-26 16:00:45.627597] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] success to create and add dag(ret=0, dag=0x7f9272926030) [2024-09-26 16:00:45.627605] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337645627600, task_id:YB42C0A8002D-0006230056D8C672-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.627613] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=7] schedule one task(task=0x7f92729263f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.627601] INFO [STORAGE] flush (ob_tx_data_memtable.cpp:917) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] [TX DATA MERGE]schedule flush tx data memtable task done(ret=0, ret="OB_SUCCESS", get_ls_id()={id:1}, this=0x7f92a2294080, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:4, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337645627600, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:45.627645] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.627650] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.627652] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726157244792838000) [2024-09-26 16:00:45.627655] INFO [STORAGE] ready_for_flush (ob_tx_data_memtable.cpp:865) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=2] memtable is frozen yet.(this=0x7f92a2294080) [2024-09-26 16:00:45.627647] INFO [MDS] flush (mds_table_mgr.cpp:172) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=18] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:45.627702] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=40] add dag success(dag=0x7f9272926560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6659188001345271846, dag_cnt=2, dag_type_cnts=1) [2024-09-26 16:00:45.627706] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9272926560) [2024-09-26 16:00:45.627709] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337645627707, task_id:YB42C0A8002D-0006230056D8C673-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.627713] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=3] schedule one task(task=0x7f9272926920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.627710] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, create_flush_dag_time:1727337645627709, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452867, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200240}, checkpoint_unit_ptr:0x7f92a1a70850}, is_freeze_clock:true}) [2024-09-26 16:00:45.627726] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=15] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337645627695, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.627745] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=14] add dag success(dag=0x7f9272926a90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-9069613569758001279, dag_cnt=3, dag_type_cnts=2) [2024-09-26 16:00:45.627747] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272926a90) [2024-09-26 16:00:45.627750] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645627749, task_id:YB42C0A8002D-0006230056D8C674-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.627749] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, create_flush_dag_time:1727337645627749, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452892, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200489}, checkpoint_unit_ptr:0x7f92a1a70080}, is_freeze_clock:true}) [2024-09-26 16:00:45.627752] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=2] schedule one task(task=0x7f9272926e50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.627753] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337645627740, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.627763] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272926fc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1288986994123435108, dag_cnt=4, dag_type_cnts=3) [2024-09-26 16:00:45.627765] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272926fc0) [2024-09-26 16:00:45.627768] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645627767, task_id:YB42C0A8002D-0006230056D8C675-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.627767] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337645627767, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452911, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847222}, checkpoint_unit_ptr:0x7f92a1a71020}, is_freeze_clock:true}) [2024-09-26 16:00:45.627770] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=2] schedule one task(task=0x7f9272927380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.627770] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337645627760, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.627782] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f92729274f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5355160120942094702, dag_cnt=5, dag_type_cnts=4) [2024-09-26 16:00:45.627784] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f92729274f0) [2024-09-26 16:00:45.627786] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337645627786, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452929, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847223}, checkpoint_unit_ptr:0x7f92a1a717f0}, is_freeze_clock:true}) [2024-09-26 16:00:45.627785] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=16][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.627791] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.627789] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337645627777, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.627794] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:45.627794] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.627798] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.627800] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:45.627801] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.627804] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.627799] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405042) [2024-09-26 16:00:45.627810] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:45.627823] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405070) [2024-09-26 16:00:45.627831] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:45.627835] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405087) [2024-09-26 16:00:45.627842] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:45.627845] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405104) [2024-09-26 16:00:45.627851] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:45.627854] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] add dag success(dag=0x7f9272927a20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1994541446894784994, dag_cnt=6, dag_type_cnts=5) [2024-09-26 16:00:45.627809] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847222}, multi_version_start=1726164780922097000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726164780922097000, old_tablet_snaphsot_version=1726164781026522000, tablet={this:0x7f92b862df20, is_inited:true, wash_score:1727337645627139000, hold_ref_cnt:true, ref_cnt:3, version:2, length:487, tablet_addr:[816017-1849-0-2305843626616520704-2](offset=1277952,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, data_tablet_id:{id:200240}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726164781225658000, v:0}, clog_checkpoint_scn:{val:1726164781225658000, v:0}, ddl_checkpoint_scn:{val:1711609699906374000, v:0}, snapshot_version:1726164781026522000, multi_version_start:1726164780922097000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726164780922097000, cur_report_version:1726164780922097000, data_checksum:3172205858, row_count:147053}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:1711609699804901001, v:0}, ddl_snapshot_version:1711609699759612000, max_sync_storage_schema_version:1711609699700488, max_serialized_medium_scn:0, ddl_execution_id:0, ddl_data_format_version:17180000769, ddl_commit_scn:{val:1711609699906374000, v:0}, mds_checkpoint_scn:{val:1726201924319685000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1711609699700488}, table_store_addr:{ptr:{this:0x7f92b862e5d8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844238649257984-0](offset=1175552,size=1145,type=2,seq=0), type:"MAJOR", tablet_id:{id:1152921504606847222}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726164780922097000, v:0}}, snapshot_version:1726164780922097000, max_merge_version:1726164780922097000}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337645627760, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843251880591360-2](offset=1245184,size=226,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276212962-0](offset=1245410,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255101817208-0](offset=1245560,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:1726164780922097000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1711609699719192000, v:0}, create_commit_version:1711609699719192000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a71020, 0x7f92a1a72790, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.627856] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272927a20) [2024-09-26 16:00:45.627858] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=48] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847222}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:45.627858] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, create_flush_dag_time:1727337645627858, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209556884, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:226}, checkpoint_unit_ptr:0x7f92a1a73f00}, is_freeze_clock:true}) [2024-09-26 16:00:45.627862] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337645627852, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.627867] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.627876] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272927f50, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8791616776033068452, dag_cnt=7, dag_type_cnts=6) [2024-09-26 16:00:45.627878] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272927f50) [2024-09-26 16:00:45.627879] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337645627879, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405141, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847408}, checkpoint_unit_ptr:0x7f92a1a746d0}, is_freeze_clock:true}) [2024-09-26 16:00:45.627880] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645627879, task_id:YB42C0A8002D-0006230056D8C676-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.627884] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=3] schedule one task(task=0x7f92729278b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.627882] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337645627869, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.627888] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337645627887, task_id:YB42C0A8002D-0006230056D8C677-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.627890] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=1] schedule one task(task=0x7f9272927de0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.627906] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.627908] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.627910] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:45.627913] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.627915] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:45.627919] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.627980] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.627984] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.627986] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:45.627991] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.628036] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.628038] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.628040] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.628042] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.628045] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:00:45.628053] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241777443612000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.628632] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272928480, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6550230011160095889, dag_cnt=8, dag_type_cnts=7) [2024-09-26 16:00:45.628647] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=14] success to create and add dag(ret=0, dag=0x7f9272928480) [2024-09-26 16:00:45.628652] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337645628652, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405158, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847409}, checkpoint_unit_ptr:0x7f92a1a75670}, is_freeze_clock:true}) [2024-09-26 16:00:45.628658] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337645627889, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628675] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=11] add dag success(dag=0x7f92729289b0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6809498982591245614, dag_cnt=9, dag_type_cnts=8) [2024-09-26 16:00:45.628678] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f92729289b0) [2024-09-26 16:00:45.628680] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, create_flush_dag_time:1727337645628680, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405174, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:200004}, checkpoint_unit_ptr:0x7f92a1a74ea0}, is_freeze_clock:true}) [2024-09-26 16:00:45.628683] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337645628670, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628693] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272928ee0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6815944285528637599, dag_cnt=10, dag_type_cnts=9) [2024-09-26 16:00:45.628695] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272928ee0) [2024-09-26 16:00:45.628697] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, create_flush_dag_time:1727337645628697, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209506214, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1}, checkpoint_unit_ptr:0x7f92a1a76610}, is_freeze_clock:true}) [2024-09-26 16:00:45.628700] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337645628690, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628709] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272929410, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-4182959058374342994, dag_cnt=11, dag_type_cnts=10) [2024-09-26 16:00:45.628711] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272929410) [2024-09-26 16:00:45.628713] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, create_flush_dag_time:1727337645628713, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405204, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:412}, checkpoint_unit_ptr:0x7f92a1a75e40}, is_freeze_clock:true}) [2024-09-26 16:00:45.628716] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337645628706, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628725] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272929940, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1586518267935340, dag_cnt=12, dag_type_cnts=11) [2024-09-26 16:00:45.628727] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272929940) [2024-09-26 16:00:45.628729] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, create_flush_dag_time:1727337645628729, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405220, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:444}, checkpoint_unit_ptr:0x7f92a1a775b0}, is_freeze_clock:true}) [2024-09-26 16:00:45.628732] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337645628723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628741] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272929e70, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3044367941103776476, dag_cnt=13, dag_type_cnts=12) [2024-09-26 16:00:45.628743] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272929e70) [2024-09-26 16:00:45.628744] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, create_flush_dag_time:1727337645628744, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455964, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:309}, checkpoint_unit_ptr:0x7f92a1a76de0}, is_freeze_clock:true}) [2024-09-26 16:00:45.628747] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337645628738, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628756] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292a3a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1596830851104282047, dag_cnt=14, dag_type_cnts=13) [2024-09-26 16:00:45.628758] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f927292a3a0) [2024-09-26 16:00:45.628759] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, create_flush_dag_time:1727337645628759, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455986, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:331}, checkpoint_unit_ptr:0x7f92a1a77d80}, is_freeze_clock:true}) [2024-09-26 16:00:45.628762] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337645628753, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628771] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292a8d0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2360195962081676673, dag_cnt=15, dag_type_cnts=14) [2024-09-26 16:00:45.628773] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f927292a8d0) [2024-09-26 16:00:45.628774] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, create_flush_dag_time:1727337645628774, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405262, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:458}, checkpoint_unit_ptr:0x7f92a1a7a490}, is_freeze_clock:true}) [2024-09-26 16:00:45.628777] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337645628768, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628786] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292ae00, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5675615614797020460, dag_cnt=16, dag_type_cnts=15) [2024-09-26 16:00:45.628788] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292ae00) [2024-09-26 16:00:45.628789] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, create_flush_dag_time:1727337645628789, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405279, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:487}, checkpoint_unit_ptr:0x7f92a1a79cc0}, is_freeze_clock:true}) [2024-09-26 16:00:45.628792] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337645628783, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628801] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292b330, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-612435342908088006, dag_cnt=17, dag_type_cnts=16) [2024-09-26 16:00:45.628803] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292b330) [2024-09-26 16:00:45.628805] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, create_flush_dag_time:1727337645628805, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405296, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:455}, checkpoint_unit_ptr:0x7f92a1a794f0}, is_freeze_clock:true}) [2024-09-26 16:00:45.628808] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337645628799, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628817] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292b860, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7868818205990873505, dag_cnt=18, dag_type_cnts=17) [2024-09-26 16:00:45.628820] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f927292b860) [2024-09-26 16:00:45.628822] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, create_flush_dag_time:1727337645628822, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405313, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:492}, checkpoint_unit_ptr:0x7f92a1a78550}, is_freeze_clock:true}) [2024-09-26 16:00:45.628825] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337645628814, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628834] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292bd90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7071731214611448811, dag_cnt=19, dag_type_cnts=18) [2024-09-26 16:00:45.628836] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292bd90) [2024-09-26 16:00:45.628838] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, create_flush_dag_time:1727337645628838, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405347, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:501}, checkpoint_unit_ptr:0x7f92a1a78d20}, is_freeze_clock:true}) [2024-09-26 16:00:45.628841] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337645628831, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628849] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292c2c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8731750006000903454, dag_cnt=20, dag_type_cnts=19) [2024-09-26 16:00:45.628853] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f927292c2c0) [2024-09-26 16:00:45.628855] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, create_flush_dag_time:1727337645628855, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405365, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:472}, checkpoint_unit_ptr:0x7f92a1a7ac60}, is_freeze_clock:true}) [2024-09-26 16:00:45.628858] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337645628847, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628867] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292c7f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6766388540859538791, dag_cnt=21, dag_type_cnts=20) [2024-09-26 16:00:45.628869] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292c7f0) [2024-09-26 16:00:45.628871] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, create_flush_dag_time:1727337645628871, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405384, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:342}, checkpoint_unit_ptr:0x7f92a1a7cba0}, is_freeze_clock:true}) [2024-09-26 16:00:45.628874] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337645628864, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628882] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292cd20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2986199386776543176, dag_cnt=22, dag_type_cnts=21) [2024-09-26 16:00:45.628884] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292cd20) [2024-09-26 16:00:45.628886] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337645628886, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405418, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:140}, checkpoint_unit_ptr:0x7f92a1a7b430}, is_freeze_clock:true}) [2024-09-26 16:00:45.628889] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337645628880, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628897] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292d250, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6947063304478012270, dag_cnt=23, dag_type_cnts=22) [2024-09-26 16:00:45.628899] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292d250) [2024-09-26 16:00:45.628901] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337645628901, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405436, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101009}, checkpoint_unit_ptr:0x7f92a1a7bc00}, is_freeze_clock:true}) [2024-09-26 16:00:45.628904] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337645628895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628913] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292d780, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5843378789243662202, dag_cnt=24, dag_type_cnts=23) [2024-09-26 16:00:45.628915] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f927292d780) [2024-09-26 16:00:45.628916] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337645628916, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405453, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101010}, checkpoint_unit_ptr:0x7f92a1a7c3d0}, is_freeze_clock:true}) [2024-09-26 16:00:45.628919] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337645628910, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628927] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292dcb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1841823684513746176, dag_cnt=25, dag_type_cnts=24) [2024-09-26 16:00:45.628929] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292dcb0) [2024-09-26 16:00:45.628931] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, create_flush_dag_time:1727337645628931, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405470, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:323}, checkpoint_unit_ptr:0x7f92a1a7d370}, is_freeze_clock:true}) [2024-09-26 16:00:45.628941] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=10] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337645628925, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628953] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927292e1e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-2678407261265885087, dag_cnt=26, dag_type_cnts=25) [2024-09-26 16:00:45.628955] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f927292e1e0) [2024-09-26 16:00:45.628957] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, create_flush_dag_time:1727337645628956, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405487, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:377}, checkpoint_unit_ptr:0x7f92a1a7e310}, is_freeze_clock:true}) [2024-09-26 16:00:45.628960] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337645628948, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628968] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292e710, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7175089551615529855, dag_cnt=27, dag_type_cnts=26) [2024-09-26 16:00:45.628970] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292e710) [2024-09-26 16:00:45.628972] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, create_flush_dag_time:1727337645628972, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405503, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:297}, checkpoint_unit_ptr:0x7f92a1a7eae0}, is_freeze_clock:true}) [2024-09-26 16:00:45.628975] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337645628966, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628984] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292ec40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3730526293242960203, dag_cnt=28, dag_type_cnts=27) [2024-09-26 16:00:45.628986] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f927292ec40) [2024-09-26 16:00:45.628987] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337645628987, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405519, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101065}, checkpoint_unit_ptr:0x7f92a1a7db40}, is_freeze_clock:true}) [2024-09-26 16:00:45.628990] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337645628981, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.628999] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292f170, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=4663792628280492528, dag_cnt=29, dag_type_cnts=28) [2024-09-26 16:00:45.629001] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292f170) [2024-09-26 16:00:45.629002] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337645629002, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405536, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:333}, checkpoint_unit_ptr:0x7f92a3bc4850}, is_freeze_clock:true}) [2024-09-26 16:00:45.629005] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337645628996, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629014] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927292f6a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8399557287176188414, dag_cnt=30, dag_type_cnts=29) [2024-09-26 16:00:45.629016] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292f6a0) [2024-09-26 16:00:45.629018] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337645629018, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405552, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101066}, checkpoint_unit_ptr:0x7f92a3bc5020}, is_freeze_clock:true}) [2024-09-26 16:00:45.629020] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337645629012, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629029] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927292fbd0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8068938909778324349, dag_cnt=31, dag_type_cnts=30) [2024-09-26 16:00:45.629031] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927292fbd0) [2024-09-26 16:00:45.629033] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, create_flush_dag_time:1727337645629033, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405569, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:334}, checkpoint_unit_ptr:0x7f92a3bc57f0}, is_freeze_clock:true}) [2024-09-26 16:00:45.629035] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337645629027, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629045] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272930100, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7665372330263152188, dag_cnt=32, dag_type_cnts=31) [2024-09-26 16:00:45.629047] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272930100) [2024-09-26 16:00:45.629048] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, create_flush_dag_time:1727337645629048, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405586, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:451}, checkpoint_unit_ptr:0x7f92a3bc4080}, is_freeze_clock:true}) [2024-09-26 16:00:45.629051] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337645629042, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629060] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272930630, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6759548225246904317, dag_cnt=33, dag_type_cnts=32) [2024-09-26 16:00:45.629062] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272930630) [2024-09-26 16:00:45.629063] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, create_flush_dag_time:1727337645629063, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405602, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:378}, checkpoint_unit_ptr:0x7f92a1a7f2b0}, is_freeze_clock:true}) [2024-09-26 16:00:45.629066] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337645629057, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629075] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272930b60, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8944614441376604790, dag_cnt=34, dag_type_cnts=33) [2024-09-26 16:00:45.629077] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272930b60) [2024-09-26 16:00:45.629078] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, create_flush_dag_time:1727337645629078, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209456003, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:121}, checkpoint_unit_ptr:0x7f92a3bc5fc0}, is_freeze_clock:true}) [2024-09-26 16:00:45.629081] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337645629072, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629090] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272931090, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=507863347806783454, dag_cnt=35, dag_type_cnts=34) [2024-09-26 16:00:45.629092] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272931090) [2024-09-26 16:00:45.629093] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, create_flush_dag_time:1727337645629093, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405633, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:215}, checkpoint_unit_ptr:0x7f92a3bc86d0}, is_freeze_clock:true}) [2024-09-26 16:00:45.629096] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337645629087, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629105] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f92729315c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1604282355669434429, dag_cnt=36, dag_type_cnts=35) [2024-09-26 16:00:45.629107] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f92729315c0) [2024-09-26 16:00:45.629108] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, create_flush_dag_time:1727337645629108, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405649, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:456}, checkpoint_unit_ptr:0x7f92a3bc6790}, is_freeze_clock:true}) [2024-09-26 16:00:45.629111] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337645629102, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629120] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272931af0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3553649416163308074, dag_cnt=37, dag_type_cnts=36) [2024-09-26 16:00:45.629122] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272931af0) [2024-09-26 16:00:45.629124] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337645629124, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405666, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:154}, checkpoint_unit_ptr:0x7f92a3bc6f60}, is_freeze_clock:true}) [2024-09-26 16:00:45.629127] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337645629117, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629136] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272932020, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8097406237500329300, dag_cnt=38, dag_type_cnts=37) [2024-09-26 16:00:45.629138] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272932020) [2024-09-26 16:00:45.629139] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337645629139, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405682, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101015}, checkpoint_unit_ptr:0x7f92a3bc7730}, is_freeze_clock:true}) [2024-09-26 16:00:45.629142] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337645629133, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629151] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272932550, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3976350754940852513, dag_cnt=39, dag_type_cnts=38) [2024-09-26 16:00:45.629153] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272932550) [2024-09-26 16:00:45.629154] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337645629154, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405699, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101016}, checkpoint_unit_ptr:0x7f92a3bc7f00}, is_freeze_clock:true}) [2024-09-26 16:00:45.629157] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337645629148, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629166] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272932a80, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8720304653345806796, dag_cnt=40, dag_type_cnts=39) [2024-09-26 16:00:45.629170] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9272932a80) [2024-09-26 16:00:45.629171] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, create_flush_dag_time:1727337645629171, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405716, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101063}, checkpoint_unit_ptr:0x7f92a3bca610}, is_freeze_clock:true}) [2024-09-26 16:00:45.629174] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337645629163, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629184] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272932fb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1830562711513853292, dag_cnt=41, dag_type_cnts=40) [2024-09-26 16:00:45.629186] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272932fb0) [2024-09-26 16:00:45.629187] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, create_flush_dag_time:1727337645629187, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405733, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:452}, checkpoint_unit_ptr:0x7f92a3bc9e40}, is_freeze_clock:true}) [2024-09-26 16:00:45.629190] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337645629180, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629200] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f92729334e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1536604874905918360, dag_cnt=42, dag_type_cnts=41) [2024-09-26 16:00:45.629213] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f92729334e0) [2024-09-26 16:00:45.629216] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337645629216, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405750, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:328}, checkpoint_unit_ptr:0x7f92a3bc8ea0}, is_freeze_clock:true}) [2024-09-26 16:00:45.629219] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337645629196, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629229] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272933a10, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6933628521298738662, dag_cnt=43, dag_type_cnts=42) [2024-09-26 16:00:45.629231] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272933a10) [2024-09-26 16:00:45.629233] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337645629233, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405767, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:330}, checkpoint_unit_ptr:0x7f92a3bc9670}, is_freeze_clock:true}) [2024-09-26 16:00:45.629236] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337645629226, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629245] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272933f40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1056199536334423564, dag_cnt=44, dag_type_cnts=43) [2024-09-26 16:00:45.629247] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272933f40) [2024-09-26 16:00:45.629248] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337645629248, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405784, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:329}, checkpoint_unit_ptr:0x7f92a3bcb5b0}, is_freeze_clock:true}) [2024-09-26 16:00:45.629251] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337645629242, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629260] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272934470, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7405463607918801432, dag_cnt=45, dag_type_cnts=44) [2024-09-26 16:00:45.629262] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272934470) [2024-09-26 16:00:45.629263] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, create_flush_dag_time:1727337645629263, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405801, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:332}, checkpoint_unit_ptr:0x7f92a3bcade0}, is_freeze_clock:true}) [2024-09-26 16:00:45.629266] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337645629257, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629275] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f92729349a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6699723950958743903, dag_cnt=46, dag_type_cnts=45) [2024-09-26 16:00:45.629277] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f92729349a0) [2024-09-26 16:00:45.629278] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, create_flush_dag_time:1727337645629278, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405818, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101067}, checkpoint_unit_ptr:0x7f92a3bcbd80}, is_freeze_clock:true}) [2024-09-26 16:00:45.629281] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337645629272, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.629287] INFO [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:655) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] traversal_flush successfully(ls_->get_ls_id()={id:1}, flush_tasks=[{table:0x7f92a1a70850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a70080, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71020, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a717f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71fc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72790, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72f60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73730, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73f00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a746d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75670, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a74ea0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76610, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75e40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a775b0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76de0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a77d80, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7a490, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a79cc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a794f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78550, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78d20, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7ac60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7cba0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7b430, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7bc00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7c3d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7d370, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7e310, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7eae0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7db40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a3bc4850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}]) [2024-09-26 16:00:45.629333] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=42] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-09-26 16:00:45.635757] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.635783] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=25] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337645627760, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:45.635792] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.635903] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=32] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:45.635909] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.635919] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847222}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:45.635945] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:1152921504606847222}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847222}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645627790, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=8.12ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.636183] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=25] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:45.636856] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=8366, total_rows=58562) [2024-09-26 16:00:45.637268] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337645627852, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.637282] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752755184, storage_schema:{this:0x7f92664ef330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828752755184, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.637300] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=12] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.637303] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.637300] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=6] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272926fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C675-0-0, dag_ret:0, dag_status:2, start_time:1727337645627770, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab470, output_block_cnt_arr:0x7f92684ab478, concurrent_cnt:1, estimate_row_cnt:329211, estimate_occupy_size:150994944, latest_update_ts:1727337645627790, estimated_finish_time:1727337662637371}) [2024-09-26 16:00:45.637313] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f9272927380, type:15, status:2, dag:{ObIDag:{this:0x7f9272926fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C675-0-0, dag_ret:0, dag_status:2, start_time:1727337645627770, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.637311] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92664ef330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:226}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752755184, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.637321] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=7] task finish process(ret=0, start_time=1727337645627780, end_time=1727337645637320, runtime=9540, *this={this:0x7f9272927380, type:15, status:2, dag:{ObIDag:{this:0x7f9272926fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C675-0-0, dag_ret:0, dag_status:2, start_time:1727337645627770, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.637332] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:226}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752755184, storage_schema:{this:0x7f92664ef330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:226}, table_count:1, [{i:0, table_key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}, merge_scn:{val:1726241777443612000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645628033, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=9.27ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.637353] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.637363] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=10] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337645627777, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:45.637370] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=31] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272927a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C677-0-0, dag_ret:0, dag_status:2, start_time:1727337645627890, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}, scanned_row_cnt_arr:0x7f92664ef480, output_block_cnt_arr:0x7f92664ef488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:56623104, latest_update_ts:1727337645628033, estimated_finish_time:1727337662637397}) [2024-09-26 16:00:45.637370] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.637378] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9272927de0, type:15, status:2, dag:{ObIDag:{this:0x7f9272927a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C677-0-0, dag_ret:0, dag_status:2, start_time:1727337645627890, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:45.637384] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=5] task finish process(ret=0, start_time=1727337645628032, end_time=1727337645637384, runtime=9352, *this={this:0x7f9272927de0, type:15, status:2, dag:{ObIDag:{this:0x7f9272927a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C677-0-0, dag_ret:0, dag_status:2, start_time:1727337645627890, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:45.637456] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:45.637462] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.637468] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847223}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:45.637499] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=25] succeed to build merge ctx(tablet_id={id:1152921504606847223}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847223}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645627903, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=9.56ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.637749] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=19] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:45.638407] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=5] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=7999, total_rows=55993) [2024-09-26 16:00:45.638582] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:45.638598] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=9] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.638670] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726158220122116000) [2024-09-26 16:00:45.638684] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=7] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.638873] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=8] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92729274f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C676-0-0, dag_ret:0, dag_status:2, start_time:1727337645627883, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92664ed470, output_block_cnt_arr:0x7f92664ed478, concurrent_cnt:1, estimate_row_cnt:332829, estimate_occupy_size:150994944, latest_update_ts:1727337645627903, estimated_finish_time:1727337662638944}) [2024-09-26 16:00:45.638887] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f92729278b0, type:15, status:2, dag:{ObIDag:{this:0x7f92729274f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C676-0-0, dag_ret:0, dag_status:2, start_time:1727337645627883, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.638893] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=5] task finish process(ret=0, start_time=1727337645627902, end_time=1727337645638892, runtime=10990, *this={this:0x7f92729278b0, type:15, status:2, dag:{ObIDag:{this:0x7f92729274f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C676-0-0, dag_ret:0, dag_status:2, start_time:1727337645627883, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.638912] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=1] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.638921] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=7] schedule one task(task=0x7f9262cbe030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.638924] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2] schedule one task(task=0x7f9268486030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.639140] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=226, range=0x7f92ab152200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d32060, count:5, allocator:0x7f92b33d87d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9287d32360, local_datums:0x7f9287d32360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:45.639944] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1, storage_schema:{this:0x7f9263b1f2b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.639973] INFO [STORAGE] prepare_tx_data_list (ob_tx_data_memtable.cpp:266) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=28] construct list more than once. skip this time(this=0x7f92a2294080) [2024-09-26 16:00:45.644971] INFO [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:329) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=3] pre-process commit versions row(get_ls_id()={id:1}, do_recycle_=true, current_time=1727337645644968, prev_recycle_time=0) [2024-09-26 16:00:45.645731] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dd8060, count:4, allocator:0x7f92636327d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287dd8350, local_datums:0x7f9287dd8350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:45.645757] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=23] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b00060, count:4, allocator:0x7f9263633558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9265b00350, local_datums:0x7f9265b00350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:45.645760] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=20][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.645826] ERROR issue_dba_error (ob_log.cpp:1875) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.645831] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.645840] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.645842] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.645845] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a620a8, hex: 0000000000000000, int: 0},idx=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a620e0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62118, hex: 58A3C56A4A280BE8, int: -1726241732082425000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-19590, column_count:10, rowkey_column_count:5, has_column_checksum:0, row_count:224, row_store_type:0, opt:0, var_column_count:0, row_offset:15527, original_length:16363, max_merged_trans_version:1726241777443612000, data_length:16363, data_zlength:16363, data_checksum:2057046089, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926fe04090, buf_size:16363, data_size:16363, row_count:224, column_count:10, max_merged_trans_version:1726241777443612000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1006468742, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16363}) [2024-09-26 16:00:45.645861] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=15][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:45.645863] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:45.645865] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:45.645866] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:45.645868] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:45.645870] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:45.645872] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9287d32360, local_datums:0x7f9287d32360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d323e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9287d32360, local_datums:0x7f9287d32360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d323e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f92b33d8118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b33d8118, limit_param:null, stmt_allocator:0x7f92b33d87d8, allocator:0x7f92b33d87d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9287d32360, local_datums:0x7f9287d32360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d323e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337645627852, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}) [2024-09-26 16:00:45.645945] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=73][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:45.645947] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9287d32360, local_datums:0x7f9287d32360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d323e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f92b33d8118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b33d8118, limit_param:null, stmt_allocator:0x7f92b33d87d8, allocator:0x7f92b33d87d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9287d32360, local_datums:0x7f9287d32360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9287d323e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337645627852, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}]) [2024-09-26 16:00:45.645982] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=35][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:45.645985] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2] partition merge iter row count(i=0, row_count=224, ghost_row_count=0, pkey={tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, table={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337645627852, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.646000] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:45.646003] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.646015] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:45.646019] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:45.646021] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337645639069, end_time=1727337645646021, runtime=6952, *this={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9272927a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C677-0-0, dag_ret:0, dag_status:2, start_time:1727337645627890, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:45.646029] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9272927a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C677-0-0, dag_ret:0, dag_status:2, start_time:1727337645627890, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.646040] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=6] dag finished(dag_ret=-4184, runtime=18149, dag_cnt=45, dag_cnts_[dag.get_type()]=44, &dag=0x7f9272927a20, dag={ObIDag:{this:0x7f9272927a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C677-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645627890, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}) [2024-09-26 16:00:45.646051] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C677-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337645627887, task_id:YB42C0A8002D-0006230056D8C677-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.646055] ERROR issue_dba_error (ob_log.cpp:1875) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.646059] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.646066] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.646068] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.646070] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9265b0c140, hex: 3230323430393131303133333438, cstr: 20240911013348},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9265b0c14e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c0e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3080, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7221, data_checksum:4284205003, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265aa6030, buf_size:7221, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1004563202, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:00:45.646083] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=13][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:45.646085] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:45.646086] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:45.646088] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:45.646090] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287dd8350, local_datums:0x7f9287dd8350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd83d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd8408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287dd8350, local_datums:0x7f9287dd8350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd83d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd8408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9263632118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263632118, limit_param:null, stmt_allocator:0x7f92636327d8, allocator:0x7f92636327d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287dd8350, local_datums:0x7f9287dd8350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd83d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd8408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:00:45.646161] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=71][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:45.646164] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=3][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287dd8350, local_datums:0x7f9287dd8350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd83d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd8408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9263632118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263632118, limit_param:null, stmt_allocator:0x7f92636327d8, allocator:0x7f92636327d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287dd8350, local_datums:0x7f9287dd8350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd83d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd8408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:00:45.646198] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=34][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:45.646201] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=2] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.646213] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=8] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337645627760, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.646225] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:45.646226] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.646239] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:45.646236] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=10] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337645627695, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=18, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:45.646243] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:45.646243] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.646244] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337645645670, end_time=1727337645646244, runtime=574, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9272926fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C675-0-0, dag_ret:0, dag_status:2, start_time:1727337645627770, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.646254] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=9][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9272926fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C675-0-0, dag_ret:0, dag_status:2, start_time:1727337645627770, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.646264] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=18493, dag_cnt=44, dag_cnts_[dag.get_type()]=43, &dag=0x7f9272926fc0, dag={ObIDag:{this:0x7f9272926fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C675-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645627770, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.646356] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C675-0-0] [lt=77] succeed to del sys task(removed_task={start_time:1727337645627767, task_id:YB42C0A8002D-0006230056D8C675-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.646442] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:45.646447] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.646437] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d32060, count:4, allocator:0x7f92636707d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:45.646455] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=15] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:4, allocator:0x7f9263671558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a62350, local_datums:0x7f9264a62350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:45.646454] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200240}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.646476] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:200240}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200240}, table_count:2, [{i:0, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8637b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645627975, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=18.48ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.646737] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=7][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.646757] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=22] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:45.646806] ERROR issue_dba_error (ob_log.cpp:1875) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.646810] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.646816] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.646819] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.646820] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9265a7a140, hex: 3230323430393131303133333131, cstr: 20240911013311},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9265a7a14e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a7a0e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a7a118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-15038, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7227, data_checksum:1452123993, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265b00030, buf_size:7227, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1818264584, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:00:45.646831] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:45.646832] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:45.646834] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:45.646835] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:45.646837] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9263670118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263670118, limit_param:null, stmt_allocator:0x7f92636707d8, allocator:0x7f92636707d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:00:45.646881] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=44][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:45.646882] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9263670118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263670118, limit_param:null, stmt_allocator:0x7f92636707d8, allocator:0x7f92636707d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:00:45.646911] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=29][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:45.646913] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=1] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.646923] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=8] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337645627777, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.646932] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:45.646941] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:45.646944] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:45.646945] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337645646390, end_time=1727337645646945, runtime=555, *this={this:0x7f9262cbe030, type:1, status:2, dag:{ObIDag:{this:0x7f92729274f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C676-0-0, dag_ret:0, dag_status:2, start_time:1727337645627883, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.646952] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262cbe030, type:1, status:2, dag:{ObIDag:{this:0x7f92729274f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C676-0-0, dag_ret:0, dag_status:2, start_time:1727337645627883, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.646959] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=19076, dag_cnt=43, dag_cnts_[dag.get_type()]=42, &dag=0x7f92729274f0, dag={ObIDag:{this:0x7f92729274f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C676-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645627883, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.646967] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C676-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337645627879, task_id:YB42C0A8002D-0006230056D8C676-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.647087] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.647096] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=8] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337645627740, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=17, max_schema_version_on_memtable=1725120002010696) [2024-09-26 16:00:45.647102] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=6] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=17, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.647154] INFO [STORAGE] get_ls_min_end_scn (ob_ls_tablet_service.cpp:5618) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=9] get ls min end scn finish(ls_checkpoint={val:1726163586617842000, v:0}) [2024-09-26 16:00:45.647164] WDIAG [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:761) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=5][errcode=-4023] failed to get snapshot version(ret=-4023, MTL_ID()=1) [2024-09-26 16:00:45.647167] INFO [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:773) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=3] get tx data recycle scn finish.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, recycle_scn={val:1726076389100953000, v:0}, min_end_scn_from_old_tablets={val:4611686018427387903, v:0}, min_end_scn_from_latest_tablets={val:1726076389100953000, v:0}) [2024-09-26 16:00:45.647173] WDIAG [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:336) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=5][errcode=-4023] get recycle ts failed.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:45.647178] WDIAG [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:236) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=2][errcode=-4023] process commit version row failed.(ret=-4023, ret="OB_EAGAIN", this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337645627600, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:45.647184] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272926560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C673-0-0, dag_ret:0, dag_status:2, start_time:1727337645627712, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5460, output_block_cnt_arr:0x7f9287db5468, concurrent_cnt:1, estimate_row_cnt:85417, estimate_occupy_size:134217728, latest_update_ts:1727337645627975, estimated_finish_time:1727337662647248}) [2024-09-26 16:00:45.647194] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=10] succeed to init merge ctx(task={this:0x7f9272926920, type:15, status:2, dag:{ObIDag:{this:0x7f9272926560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C673-0-0, dag_ret:0, dag_status:2, start_time:1727337645627712, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.647189] INFO [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:254) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=11] [TX DATA MERGE]pre process for merge done(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}, this=0x7f92a2294080, tg=time guard have no click events for optimization, fake_tx_data_guard={tx_data:{tx_id:{txid:9223372036854775807}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337645627600, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:45.647201] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1887, total_rows=92463) [2024-09-26 16:00:45.647201] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=7] task finish process(ret=0, start_time=1727337645627973, end_time=1727337645647201, runtime=19228, *this={this:0x7f9272926920, type:15, status:2, dag:{ObIDag:{this:0x7f9272926560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C673-0-0, dag_ret:0, dag_status:2, start_time:1727337645627712, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.647205] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.647202] WDIAG [STORAGE.COMPACTION] pre_process_tx_data_table_merge_ (ob_tx_table_merge_task.cpp:117) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=12][errcode=-4023] do pre process for tx data table merge failed.(ret=-4023, ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, table={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337645627600, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:45.647209] WDIAG [STORAGE.COMPACTION] inner_init_ctx (ob_tx_table_merge_task.cpp:157) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=7][errcode=-4023] pre process tx data table for merge failed.(ret=-4023, ret="OB_EAGAIN", ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.647211] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200489}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:19, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.647213] WDIAG [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1061) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=3][errcode=-4023] fail to inner init ctx(ret=-4023, tablet_id={id:49402}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, create_snapshot_version:0, is_full_merge:false, merge_level:1, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:4, range_array:[], concurrent_cnt:0, is_inited:false}, schema_ctx:{schema_version:1, storage_schema:{this:0x7f9263b1f2b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:49402}, table_count:1, [{i:0, table_key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref:4}]}, schedule_major:false, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}, merge_scn:{val:4611686018427387903, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:NULL, compaction_filter:NULL, time_guard:total=19.57ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}) [2024-09-26 16:00:45.647230] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:975) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=17][errcode=-4023] failed to build merge ctx(ret=-4023, ctx->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.647233] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:994) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=3][errcode=-4023] sstable merge finish(ret=-4023, ctx=0x7f9263b1e060, task={this:0x7f92729263f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272926030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C672-0-0, dag_ret:0, dag_status:2, start_time:1727337645627612, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:45.647239] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=6][errcode=-4023] failed to process task(ret=-4023) [2024-09-26 16:00:45.647241] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=2] task finish process(ret=-4023, start_time=1727337645627634, end_time=1727337645647241, runtime=19607, *this={this:0x7f92729263f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272926030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C672-0-0, dag_ret:0, dag_status:2, start_time:1727337645627612, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:45.647230] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:200489}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200489}, table_count:2, [{i:0, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645627778, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=19.43ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.647246] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=5][errcode=-4023] failed to do work(ret=-4023, *task_={this:0x7f92729263f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272926030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C672-0-0, dag_ret:0, dag_status:2, start_time:1727337645627612, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.647253] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=5] dag finished(dag_ret=-4023, runtime=19641, dag_cnt=42, dag_cnts_[dag.get_type()]=0, &dag=0x7f9272926030, dag={ObIDag:{this:0x7f9272926030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C672-0-0, dag_ret:-4023, dag_status:5, start_time:1727337645627612, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}) [2024-09-26 16:00:45.647261] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C672-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645627600, task_id:YB42C0A8002D-0006230056D8C672-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.647358] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=1] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.647368] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337645647365, task_id:YB42C0A8002D-0006230056D8C678-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.647371] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=3] schedule one task(task=0x7f9272928310, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.647377] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645647376, task_id:YB42C0A8002D-0006230056D8C679-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.647379] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=2] schedule one task(task=0x7f9272928840, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.647397] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645647395, task_id:YB42C0A8002D-0006230056D8C67A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.647400] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=2] schedule one task(task=0x7f9272928d70, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.647405] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645647404, task_id:YB42C0A8002D-0006230056D8C67B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.647407] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=2] schedule one task(task=0x7f92729292a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.647431] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.647435] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.647438] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.647445] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.647474] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.647477] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.647478] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.647481] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.647486] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.647512] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=18] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.647533] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.647536] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.647537] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.647540] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.647543] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861eec0, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:493, tablet_addr:[816017-1849-0-2305843633059586048-2](offset=1892352,size=581,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1}, data_tablet_id:{id:1}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:778798285, row_count:7249}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861f59c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844357835309056-0](offset=1884160,size=1256,type=2,seq=0), type:"MAJOR", tablet_id:{id:1}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077591663360001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843964845802728-0](offset=1885416,size=890,type=2,seq=0), type:"MINOR", tablet_id:{id:1}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178578410947000, upper_trans_version:1726178578410947000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337645628690, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcd4f0, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241621541056000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405180}, this:0x7f92a3bcd4f0, timestamp:1727334209405180, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:53, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277807268534000, v:0}, rec_scn:{val:1726241631569302000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771609600-2](offset=1888256,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843169203114317-0](offset=1888589,size=149,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881234914-0](offset=1888738,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828751995331000, v:0}, create_commit_version:1710828751995331000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a76610, 0x7f92a3bcd4f0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.647589] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=46] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.647593] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241621541056000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.647611] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272926a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C674-0-0, dag_ret:0, dag_status:2, start_time:1727337645627752, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515460, output_block_cnt_arr:0x7f9268515468, concurrent_cnt:1, estimate_row_cnt:254187, estimate_occupy_size:138412032, latest_update_ts:1727337645627778, estimated_finish_time:1727337662647676}) [2024-09-26 16:00:45.647621] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=10] succeed to init merge ctx(task={this:0x7f9272926e50, type:15, status:2, dag:{ObIDag:{this:0x7f9272926a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C674-0-0, dag_ret:0, dag_status:2, start_time:1727337645627752, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.647627] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=6] task finish process(ret=0, start_time=1727337645627774, end_time=1727337645647627, runtime=19853, *this={this:0x7f9272926e50, type:15, status:2, dag:{ObIDag:{this:0x7f9272926a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C674-0-0, dag_ret:0, dag_status:2, start_time:1727337645627752, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.647659] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=2] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.647721] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92aaed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264b20060, count:3, allocator:0x7f92a31fa7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264b20320, local_datums:0x7f9264b20320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:45.647724] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:3, allocator:0x7f92a31fc7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a3a320, local_datums:0x7f9263a3a320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:45.647739] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=15] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92aaed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a66060, count:3, allocator:0x7f92a31fb558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a66320, local_datums:0x7f9264a66320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:45.647740] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=13] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:3, allocator:0x7f92a31fd558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:45.647896] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.647937] ERROR issue_dba_error (ob_log.cpp:1875) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.647941] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=7][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.647940] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.647947] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.647949] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.647951] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d36070, hex: 9F094D0300000000, int: 55380383},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d360a8, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d360e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3044, column_count:19, rowkey_column_count:3, has_column_checksum:0, row_count:83, row_store_type:0, opt:5, var_column_count:0, row_offset:16109, original_length:16381, max_merged_trans_version:1726163995623997000, data_length:16381, data_zlength:7759, data_checksum:3416849541, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265b00030, buf_size:7759, data_size:16381, row_count:83, column_count:19, max_merged_trans_version:1726163995623997000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3537734793, row_count_delta:83, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16381}) [2024-09-26 16:00:45.647962] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:45.647963] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:45.647965] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:45.647966] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:45.647978] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.647981] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf2041c8 0xf217c49 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.647987] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.647989] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.647991] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9264d60108, hex: 313030303030333830, cstr: 100000380},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d600a8, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d600e0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-29413, column_count:20, rowkey_column_count:3, has_column_checksum:0, row_count:100, row_store_type:0, opt:0, var_column_count:0, row_offset:16082, original_length:16422, max_merged_trans_version:1726189672100745000, data_length:16454, data_zlength:5272, data_checksum:4171874375, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92729ee030, buf_size:5272, data_size:16454, row_count:100, column_count:20, max_merged_trans_version:1726189672100745000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:371002229, row_count_delta:22, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16422}) [2024-09-26 16:00:45.648001] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=10][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:45.648003] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:45.648004] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:45.648005] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:45.647968] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a66320, local_datums:0x7f9264a66320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a663a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a664f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a665d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a666e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a66320, local_datums:0x7f9264a66320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a663a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a664f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a665d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a666e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f92a31fae98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31fae98, limit_param:null, stmt_allocator:0x7f92a31fb558, allocator:0x7f92a31fb558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1073741824, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a66320, local_datums:0x7f9264a66320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a663a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a664f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a665d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a666e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337645627740, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}) [2024-09-26 16:00:45.648028] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=60][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:45.648030] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a66320, local_datums:0x7f9264a66320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a663a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a664f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a665d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a666e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f92a31fae98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31fae98, limit_param:null, stmt_allocator:0x7f92a31fb558, allocator:0x7f92a31fb558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1073741824, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264a66320, local_datums:0x7f9264a66320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a663a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a664f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a665d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264a666e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337645627740, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}]) [2024-09-26 16:00:45.648067] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=37][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:45.648069] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=2] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.648080] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=9] partition merge iter row count(i=1, row_count=83, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, table={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337645627740, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.648089] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:45.648097] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:45.648007] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1526) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, partition_fuser_->get_result_row()={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a38148, local_datums:0x7f9263a38148}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38190, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a381c8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a383c0, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a383f8, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, fuse_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a3a320, local_datums:0x7f9263a3a320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92a31fc118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31fc118, limit_param:null, stmt_allocator:0x7f92a31fc7d8, allocator:0x7f92a31fc7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a3a320, local_datums:0x7f9263a3a320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92a31fce98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31fce98, limit_param:null, stmt_allocator:0x7f92a31fd558, allocator:0x7f92a31fd558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337645627695, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:45.648100] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:45.648101] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337645647672, end_time=1727337645648101, runtime=429, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9272926a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C674-0-0, dag_ret:0, dag_status:2, start_time:1727337645627752, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.648108] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9272926a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C674-0-0, dag_ret:0, dag_status:2, start_time:1727337645627752, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.648116] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=20363, dag_cnt=41, dag_cnts_[dag.get_type()]=41, &dag=0x7f9272926a90, dag={ObIDag:{this:0x7f9272926a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C674-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645627752, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.648123] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C674-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645627749, task_id:YB42C0A8002D-0006230056D8C674-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.648149] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645648147, task_id:YB42C0A8002D-0006230056D8C67C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.648152] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=3] schedule one task(task=0x7f92729297d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.648100] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=93][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a3a320, local_datums:0x7f9263a3a320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92a31fc118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31fc118, limit_param:null, stmt_allocator:0x7f92a31fc7d8, allocator:0x7f92a31fc7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a3a320, local_datums:0x7f9263a3a320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92a31fce98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31fce98, limit_param:null, stmt_allocator:0x7f92a31fd558, allocator:0x7f92a31fd558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a3e320, local_datums:0x7f9263a3e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337645627695, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:45.648193] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=92][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:45.648195] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=2] partition merge iter row count(i=0, row_count=23, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.648205] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=8] partition merge iter row count(i=1, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337645627695, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.648214] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:45.648225] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:45.648228] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:45.648229] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337645647674, end_time=1727337645648229, runtime=555, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9272926560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C673-0-0, dag_ret:0, dag_status:2, start_time:1727337645627712, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.648236] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9272926560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C673-0-0, dag_ret:0, dag_status:2, start_time:1727337645627712, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.648242] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=20530, dag_cnt=40, dag_cnts_[dag.get_type()]=40, &dag=0x7f9272926560, dag={ObIDag:{this:0x7f9272926560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C673-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645627712, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.648250] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C673-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337645627707, task_id:YB42C0A8002D-0006230056D8C673-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.648265] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645648264, task_id:YB42C0A8002D-0006230056D8C67D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.648267] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=2] schedule one task(task=0x7f9272929d00, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.648296] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.648300] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.648302] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.648308] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.648342] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.648344] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.648346] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.648348] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.648352] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726176799210367000, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b862fe60, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:497, tablet_addr:[816017-1849-0-2305843637353291776-2](offset=630784,size=585,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:412}, data_tablet_id:{id:412}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726176799210367000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3559841642, row_count:4}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8630518, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844255828574208-0](offset=622592,size=1161,type=2,seq=0), type:"MAJOR", tablet_id:{id:412}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725843390668168000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335631832201-0](offset=623753,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:412}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337645628706, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd6790, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334211402085}, this:0x7f92a3bd6790, timestamp:1727334211402085, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:70, clock:618659840}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277807335692000, v:0}, rec_scn:{val:1726250403931267000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365696606208-2](offset=626688,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170275594572-0](offset=627020,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251879973346-0](offset=627170,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216446493380-0](offset=627396,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a75e40, 0x7f92a3bd6790, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.648394] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=42] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.648403] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.648433] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.648435] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.648437] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.648440] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.648442] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.648447] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.648544] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:45.648553] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.649743] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337645628690, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.649753] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:0, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.649776] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=18] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.649779] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.649784] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:0, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.649804] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:1}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:0, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1}, table_count:1, [{i:0, table_key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}, merge_scn:{val:1726241621541056000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645647529, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.25ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.649828] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272928ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67B-0-0, dag_ret:0, dag_status:2, start_time:1727337645647407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}, scanned_row_cnt_arr:0x7f92685e7480, output_block_cnt_arr:0x7f92685e7488, concurrent_cnt:1, estimate_row_cnt:3, estimate_occupy_size:6291456, latest_update_ts:1727337645647529, estimated_finish_time:1727337662649831}) [2024-09-26 16:00:45.649834] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f92729292a0, type:15, status:2, dag:{ObIDag:{this:0x7f9272928ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67B-0-0, dag_ret:0, dag_status:2, start_time:1727337645647407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:45.649839] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=5] task finish process(ret=0, start_time=1727337645647528, end_time=1727337645649839, runtime=2311, *this={this:0x7f92729292a0, type:15, status:2, dag:{ObIDag:{this:0x7f9272928ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67B-0-0, dag_ret:0, dag_status:2, start_time:1727337645647407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:45.649852] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=2] schedule one task(task=0x7f9268486030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.649933] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:8}, cols_desc:{ObIArray:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:5, allocator:0x7f92865b67d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5a360, local_datums:0x7f9263b5a360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:45.650318] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=16][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.650360] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.650366] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.650372] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.650375] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.650376] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 17, flag: 0, null: 0, ptr: 0x7f9263b5e178, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e0a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 15, flag: 0, null: 0, ptr: 0x7f9263b5e189, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e118, hex: 60D2640CCC290BE8, int: -1726240075808452000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-20032, column_count:8, rowkey_column_count:5, has_column_checksum:0, row_count:158, row_store_type:0, opt:0, var_column_count:0, row_offset:15849, original_length:16421, max_merged_trans_version:1726241621541056000, data_length:16421, data_zlength:16421, data_checksum:3599888316, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6e04090, buf_size:16421, data_size:16421, row_count:158, column_count:8, max_merged_trans_version:1726241621541056000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1714289658, row_count_delta:2, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16421}) [2024-09-26 16:00:45.650388] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:45.650390] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=1][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:45.650391] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:45.650393] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:45.650394] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:45.650396] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:45.650397] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5a360, local_datums:0x7f9263b5a360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5a360, local_datums:0x7f9263b5a360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f92865b6118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92865b6118, limit_param:null, stmt_allocator:0x7f92865b67d8, allocator:0x7f92865b67d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5a360, local_datums:0x7f9263b5a360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337645628690, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:45.650478] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=81][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:45.650480] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5a360, local_datums:0x7f9263b5a360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f92865b6118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92865b6118, limit_param:null, stmt_allocator:0x7f92865b67d8, allocator:0x7f92865b67d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5a360, local_datums:0x7f9263b5a360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337645628690, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:45.650516] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=36][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:45.650518] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=1] partition merge iter row count(i=0, row_count=158, ghost_row_count=0, pkey={tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, table={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337645628690, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.650530] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:45.650541] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=4][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:45.650554] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=13][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:45.650556] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337645649873, end_time=1727337645650556, runtime=683, *this={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9272928ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67B-0-0, dag_ret:0, dag_status:2, start_time:1727337645647407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:45.650562] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9272928ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67B-0-0, dag_ret:0, dag_status:2, start_time:1727337645647407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.650570] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=3162, dag_cnt=39, dag_cnts_[dag.get_type()]=39, &dag=0x7f9272928ee0, dag={ObIDag:{this:0x7f9272928ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645647407, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}) [2024-09-26 16:00:45.650577] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645647404, task_id:YB42C0A8002D-0006230056D8C67B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.650593] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] succeed to add sys task(task={start_time:1727337645650590, task_id:YB42C0A8002D-0006230056D8C67E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.650600] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=6] schedule one task(task=0x7f927292a230, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.650713] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:45.650815] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=51, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:45.651225] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [2925][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=12] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f92b77d02b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-09-26 16:00:45.651478] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=346027, clean_num=31457) [2024-09-26 16:00:45.651512] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.651516] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.651518] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.651522] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.651525] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.651532] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241316899724000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.651566] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337645627889, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.651576] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=10] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:45.651579] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9263b1f480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.651606] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:45.651620] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=9] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.651703] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3223, total_rows=22561) [2024-09-26 16:00:45.651710] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=6] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.651723] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263b1f480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847409}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.651742] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:1152921504606847409}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9263b1f480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847409}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8640e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645647421, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.29ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.651853] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272928480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C679-0-0, dag_ret:0, dag_status:2, start_time:1727337645647379, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263b1f5d0, output_block_cnt_arr:0x7f9263b1f5d8, concurrent_cnt:1, estimate_row_cnt:32439, estimate_occupy_size:18874368, latest_update_ts:1727337645647421, estimated_finish_time:1727337662651862}) [2024-09-26 16:00:45.651860] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9272928840, type:15, status:2, dag:{ObIDag:{this:0x7f9272928480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C679-0-0, dag_ret:0, dag_status:2, start_time:1727337645647379, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.651868] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=7] task finish process(ret=0, start_time=1727337645647409, end_time=1727337645651867, runtime=4458, *this={this:0x7f9272928840, type:15, status:2, dag:{ObIDag:{this:0x7f9272928480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C679-0-0, dag_ret:0, dag_status:2, start_time:1727337645647379, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.651884] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=3] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.651971] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=13] multi version scan iterator init succ(param.table_id_=1152921504606847409, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ee060, count:5, allocator:0x7f92b33fe7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664ee360, local_datums:0x7f92664ee360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:45.652182] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=12][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.652220] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.652223] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.652230] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.652232] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=1][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.652233] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 7, flag: 0, null: 0, ptr: 0x7f9262c6e178, hex: 31303033343330, cstr: 1003430},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9262c6e17f, hex: 3230323430393133303435303537, cstr: 20240913045057},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e0e0, hex: 6E48500300000000, int: 55593070},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e118, hex: 58B03BC5A1610BE8, int: -1726178684739801000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-26357, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:250, row_store_type:0, opt:5, var_column_count:0, row_offset:15394, original_length:16334, max_merged_trans_version:1726190643638412000, data_length:16334, data_zlength:5786, data_checksum:2846467531, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92684de030, buf_size:5786, data_size:16334, row_count:250, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:796410710, row_count_delta:250, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16334}) [2024-09-26 16:00:45.652244] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:45.652246] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:45.652247] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:45.652248] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:45.652250] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664ee360, local_datums:0x7f92664ee360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664ee360, local_datums:0x7f92664ee360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92b33fe118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b33fe118, limit_param:null, stmt_allocator:0x7f92b33fe7d8, allocator:0x7f92b33fe7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:94314390480696, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664ee360, local_datums:0x7f92664ee360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337645627889, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:45.652318] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=68][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:45.652320] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664ee360, local_datums:0x7f92664ee360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92b33fe118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b33fe118, limit_param:null, stmt_allocator:0x7f92b33fe7d8, allocator:0x7f92b33fe7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:94314390480696, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664ee360, local_datums:0x7f92664ee360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337645627889, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:45.652352] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=31][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:45.652354] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=1] partition merge iter row count(i=0, row_count=250, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337645627889, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.652365] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:45.652381] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:45.652386] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:45.652387] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337645651928, end_time=1727337645652387, runtime=459, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272928480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C679-0-0, dag_ret:0, dag_status:2, start_time:1727337645647379, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.652394] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272928480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C679-0-0, dag_ret:0, dag_status:2, start_time:1727337645647379, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.652401] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=5022, dag_cnt=38, dag_cnts_[dag.get_type()]=38, &dag=0x7f9272928480, dag={ObIDag:{this:0x7f9272928480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C679-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645647379, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.652411] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C679-0-0] [lt=8] succeed to del sys task(removed_task={start_time:1727337645647376, task_id:YB42C0A8002D-0006230056D8C679-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.652429] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337645652427, task_id:YB42C0A8002D-0006230056D8C67F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.652434] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=4] schedule one task(task=0x7f927292a760, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.652531] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.652538] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.652540] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.652545] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.652549] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.652557] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241465393532000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.652586] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337645628723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.652589] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337645628670, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.652599] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828761605424, storage_schema:{this:0x7f9265a7b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828761605424, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.652599] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753327496, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753327496, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.652610] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.652613] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.652614] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.652617] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.652619] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9265a7b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200004}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1710828761605424, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.652636] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:200004}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828761605424, storage_schema:{this:0x7f9265a7b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200004}, table_count:1, [{i:0, table_key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8621da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645647471, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.15ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.652622] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:444}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753327496, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.652657] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92729289b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67A-0-0, dag_ret:0, dag_status:2, start_time:1727337645647400, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265a7b460, output_block_cnt_arr:0x7f9265a7b468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337645647471, estimated_finish_time:1727337662652659}) [2024-09-26 16:00:45.652663] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272928d70, type:15, status:2, dag:{ObIDag:{this:0x7f92729289b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67A-0-0, dag_ret:0, dag_status:2, start_time:1727337645647400, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.652668] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=4] task finish process(ret=0, start_time=1727337645647470, end_time=1727337645652668, runtime=5198, *this={this:0x7f9272928d70, type:15, status:2, dag:{ObIDag:{this:0x7f92729289b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67A-0-0, dag_ret:0, dag_status:2, start_time:1727337645647400, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.652676] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=47] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.652668] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=39] succeed to build merge ctx(tablet_id={id:444}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753327496, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:444}, table_count:1, [{i:0, table_key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8632d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645648429, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.19ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.652703] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=28] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272929940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67D-0-0, dag_ret:0, dag_status:2, start_time:1727337645648267, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3490, output_block_cnt_arr:0x7f92728f3498, concurrent_cnt:1, estimate_row_cnt:4, estimate_occupy_size:6291456, latest_update_ts:1727337645648429, estimated_finish_time:1727337662652706}) [2024-09-26 16:00:45.652711] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9272929d00, type:15, status:2, dag:{ObIDag:{this:0x7f9272929940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67D-0-0, dag_ret:0, dag_status:2, start_time:1727337645648267, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.652716] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=5] task finish process(ret=0, start_time=1727337645648427, end_time=1727337645652716, runtime=4289, *this={this:0x7f9272929d00, type:15, status:2, dag:{ObIDag:{this:0x7f9272929940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67D-0-0, dag_ret:0, dag_status:2, start_time:1727337645648267, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.652725] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=3] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.652779] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=200004, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264b20060, count:3, allocator:0x7f92864e07d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264b20320, local_datums:0x7f9264b20320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:45.652947] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=12][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.652987] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.652990] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.652996] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.652998] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.653000] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a66070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a660a8, hex: C852A9F488570BE8, int: -1726189786434547000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a660e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19071, column_count:8, rowkey_column_count:3, has_column_checksum:0, row_count:218, row_store_type:0, opt:0, var_column_count:0, row_offset:15549, original_length:16361, max_merged_trans_version:1726190661653507000, data_length:16361, data_zlength:5996, data_checksum:2604580500, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9287d2e030, buf_size:5996, data_size:16361, row_count:218, column_count:8, max_merged_trans_version:1726190661653507000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:83488362, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16361}) [2024-09-26 16:00:45.653010] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=10][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:45.653012] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:45.653013] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:45.653015] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:45.653016] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264b20320, local_datums:0x7f9264b20320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b203a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264b20320, local_datums:0x7f9264b20320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b203a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f92864e0118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92864e0118, limit_param:null, stmt_allocator:0x7f92864e07d8, allocator:0x7f92864e07d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4199973039878850632, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264b20320, local_datums:0x7f9264b20320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b203a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337645628670, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:45.653071] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=55][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:45.653072] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264b20320, local_datums:0x7f9264b20320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b203a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f92864e0118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92864e0118, limit_param:null, stmt_allocator:0x7f92864e07d8, allocator:0x7f92864e07d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4199973039878850632, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264b20320, local_datums:0x7f9264b20320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b203a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b20480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337645628670, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:45.653107] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=35][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:45.653109] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=2] partition merge iter row count(i=0, row_count=218, ghost_row_count=0, pkey={tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337645628670, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.653121] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:45.653130] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:45.653133] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:45.653135] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337645652744, end_time=1727337645653135, runtime=391, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f92729289b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67A-0-0, dag_ret:0, dag_status:2, start_time:1727337645647400, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.653141] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f92729289b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67A-0-0, dag_ret:0, dag_status:2, start_time:1727337645647400, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.653148] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=5748, dag_cnt=37, dag_cnts_[dag.get_type()]=37, &dag=0x7f92729289b0, dag={ObIDag:{this:0x7f92729289b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645647400, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.653155] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67A-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645647395, task_id:YB42C0A8002D-0006230056D8C67A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.653186] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645653184, task_id:YB42C0A8002D-0006230056D8C680-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.653189] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=3] schedule one task(task=0x7f927292ac90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.653211] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.653215] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.653217] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.653222] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.653234] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=444, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b56060, count:6, allocator:0x7f92865547d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9263b56390, local_datums:0x7f9263b56390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:45.653226] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8627b60, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:523, tablet_addr:[816017-1849-0-2305843665271992320-2](offset=2043904,size=611,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:458}, data_tablet_id:{id:458}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2897668175, row_count:77572}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86282a8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844621975949312-0](offset=2035712,size=1502,type=2,seq=0), type:"MAJOR", tablet_id:{id:458}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389055265000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844927992370654-0](offset=2037214,size=1787,type=2,seq=0), type:"MINOR", tablet_id:{id:458}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178190631204000, upper_trans_version:1726178190631204000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337645628768, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd1b40, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209496526}, this:0x7f92a3bd1b40, timestamp:1727334209496526, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:61, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274225933800000, v:0}, rec_scn:{val:1726241806895572002, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771761152-2](offset=2039808,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046811469-0](offset=2040141,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881386491-0](offset=2040315,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447906525-0](offset=2040541,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7a490, 0x7f92a3bd1b40, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.653269] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=43] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.653276] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.653427] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=13][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.653461] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.653464] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.653470] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.653472] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.653474] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e0a8, hex: 0300000000000000, int: 3},idx=2:{len: 12, flag: 0, null: 0, ptr: 0x7f9263b5e1b0, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e118, hex: 420B000000000000, int: 2882},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e150, hex: 288E681FE7280BE8, int: -1726241059036951000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e188, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:24981, column_count:9, rowkey_column_count:6, has_column_checksum:0, row_count:207, row_store_type:0, opt:0, var_column_count:0, row_offset:15610, original_length:16378, max_merged_trans_version:1726241059036951000, data_length:16378, data_zlength:16378, data_checksum:1574137226, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926fc04090, buf_size:16378, data_size:16378, row_count:207, column_count:9, max_merged_trans_version:1726241059036951000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:666643265, row_count_delta:3, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16378}) [2024-09-26 16:00:45.653484] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=10][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:45.653486] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:45.653487] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:45.653489] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=2][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:45.653490] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:45.653492] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:45.653493] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9263b56390, local_datums:0x7f9263b56390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b564b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9263b56390, local_datums:0x7f9263b56390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b564b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f9286554118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286554118, limit_param:null, stmt_allocator:0x7f92865547d8, allocator:0x7f92865547d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9263b56390, local_datums:0x7f9263b56390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b564b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337645628723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}) [2024-09-26 16:00:45.653556] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=63][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:45.653557] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9263b56390, local_datums:0x7f9263b56390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b564b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f9286554118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286554118, limit_param:null, stmt_allocator:0x7f92865547d8, allocator:0x7f92865547d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9263b56390, local_datums:0x7f9263b56390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b563d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b564b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b56560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337645628723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}]) [2024-09-26 16:00:45.653595] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=38][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:45.653597] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=2] partition merge iter row count(i=0, row_count=207, ghost_row_count=0, pkey={tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337645628723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.653608] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:45.653616] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:45.653619] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:45.653620] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337645653197, end_time=1727337645653620, runtime=423, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272929940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67D-0-0, dag_ret:0, dag_status:2, start_time:1727337645648267, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.653627] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272929940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67D-0-0, dag_ret:0, dag_status:2, start_time:1727337645648267, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.653633] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=5366, dag_cnt=36, dag_cnts_[dag.get_type()]=36, &dag=0x7f9272929940, dag={ObIDag:{this:0x7f9272929940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645648267, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.653640] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67D-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645648264, task_id:YB42C0A8002D-0006230056D8C67D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.653655] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645653653, task_id:YB42C0A8002D-0006230056D8C681-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.653658] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=3] schedule one task(task=0x7f927292b1c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.653667] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337645628706, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.653676] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753247280, storage_schema:{this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828753247280, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.653684] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.653686] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.653688] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.653689] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.653690] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.653694] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.653694] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:412}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753247280, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.653710] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:412}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753247280, storage_schema:{this:0x7f92685eb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:412}, table_count:1, [{i:0, table_key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645648338, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.35ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.653730] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272929410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67C-0-0, dag_ret:0, dag_status:2, start_time:1727337645648152, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685eb470, output_block_cnt_arr:0x7f92685eb478, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337645648338, estimated_finish_time:1727337662653733}) [2024-09-26 16:00:45.653697] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b863cf80, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:522, tablet_addr:[816017-1849-0-2305843664198197248-2](offset=1990656,size=610,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:487}, data_tablet_id:{id:487}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1549633270, row_count:11693}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863d6c8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844647745695744-0](offset=1978368,size=1526,type=2,seq=0), type:"MAJOR", tablet_id:{id:487}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389060348000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843977730799094-0](offset=1979894,size=902,type=2,seq=0), type:"MINOR", tablet_id:{id:487}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178190844721000, upper_trans_version:1726178190844721000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337645628783, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd4850, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209506557}, this:0x7f92a3bd4850, timestamp:1727334209506557, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:66, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274231198751000, v:0}, rec_scn:{val:1726241813664082000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843367845449728-2](offset=1986560,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046758222-0](offset=1986894,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881333244-0](offset=1987068,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447853278-0](offset=1987294,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a79cc0, 0x7f92a3bd4850, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.653736] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=39] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.653735] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f92729297d0, type:15, status:2, dag:{ObIDag:{this:0x7f9272929410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67C-0-0, dag_ret:0, dag_status:2, start_time:1727337645648152, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.653740] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=5] task finish process(ret=0, start_time=1727337645648337, end_time=1727337645653740, runtime=5403, *this={this:0x7f92729297d0, type:15, status:2, dag:{ObIDag:{this:0x7f9272929410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67C-0-0, dag_ret:0, dag_status:2, start_time:1727337645648152, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.653743] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.653747] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=3] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.653804] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=412, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:7}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265a74060, count:4, allocator:0x7f92636287d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9265a74350, local_datums:0x7f9265a74350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:45.653828] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.653862] ERROR issue_dba_error (ob_log.cpp:1875) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.653865] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.653871] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.653873] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.653874] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 23, flag: 0, null: 0, ptr: 0x7f9264a62140, hex: 6865617274626561745F736572766963655F65706F6368, cstr: heartbeat_service_epoch},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a620e0, hex: 8808AF9C9C5C0BE8, int: -1726184204453083000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-7627, column_count:7, rowkey_column_count:4, has_column_checksum:0, row_count:1, row_store_type:0, opt:5, var_column_count:0, row_offset:136, original_length:80, max_merged_trans_version:1726184204453083000, data_length:80, data_zlength:80, data_checksum:1549639079, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92b6a04090, buf_size:80, data_size:80, row_count:1, column_count:7, max_merged_trans_version:1726184204453083000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:37835160, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:80}) [2024-09-26 16:00:45.653886] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=11][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:45.653887] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=1][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:00:45.653889] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=2][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:00:45.653891] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=1][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:00:45.653892] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=1][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:00:45.653894] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337645628706, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.653906] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=10][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:45.653913] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:45.653916] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:45.653918] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337645653772, end_time=1727337645653918, runtime=146, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9272929410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67C-0-0, dag_ret:0, dag_status:2, start_time:1727337645648152, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.653924] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9272929410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67C-0-0, dag_ret:0, dag_status:2, start_time:1727337645648152, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.653930] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=5778, dag_cnt=35, dag_cnts_[dag.get_type()]=35, &dag=0x7f9272929410, dag={ObIDag:{this:0x7f9272929410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645648152, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.653937] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67C-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645648147, task_id:YB42C0A8002D-0006230056D8C67C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.653950] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645653949, task_id:YB42C0A8002D-0006230056D8C682-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.653952] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=2] schedule one task(task=0x7f927292b6f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.653989] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337645320844]}) [2024-09-26 16:00:45.654009] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.654012] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.654014] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.654015] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855E6-0-0] [lt=8][errcode=-4006] not init(ret=-4006) [2024-09-26 16:00:45.654017] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.654022] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:556) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855E6-0-0] [lt=4] finish scan map holding items(scan_cnt=0) [2024-09-26 16:00:45.654038] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855E7-0-0] [lt=2] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:45.654021] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861cf80, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:528, tablet_addr:[816017-1849-0-2305843670640488448-2](offset=1830912,size=616,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:455}, data_tablet_id:{id:455}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4242650745, row_count:15909}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861d6ec, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305845137371811840-0](offset=1822720,size=1982,type=2,seq=0), type:"MAJOR", tablet_id:{id:455}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389074974000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305845058988660670-0](offset=1824702,size=1909,type=2,seq=0), type:"MINOR", tablet_id:{id:455}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178192070487000, upper_trans_version:1726178192070487000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337645628799, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcec60, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209461810}, this:0x7f92a3bcec60, timestamp:1727334209461810, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:55, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274239678928000, v:0}, rec_scn:{val:1726241818022255001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771548160-2](offset=1826816,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843202489049421-0](offset=1827149,size=180,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881173505-0](offset=1827329,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447693539-0](offset=1827555,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a794f0, 0x7f92a3bcec60, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.654060] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=38] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.654066] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.655207] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21] success to notify_need_writing_throttling True [2024-09-26 16:00:45.655244] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.655257] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337645628738, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.655267] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752918752, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828752918752, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.655281] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.655294] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.655301] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:309}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752918752, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.655331] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=24] succeed to build merge ctx(tablet_id={id:309}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752918752, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:309}, table_count:1, [{i:0, table_key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}, merge_scn:{val:1726241316899724000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8631da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645651506, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.79ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.655354] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272929e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67E-0-0, dag_ret:0, dag_status:2, start_time:1727337645650599, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1490, output_block_cnt_arr:0x7f92728f1498, concurrent_cnt:1, estimate_row_cnt:6, estimate_occupy_size:6291456, latest_update_ts:1727337645651506, estimated_finish_time:1727337662655357}) [2024-09-26 16:00:45.655360] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292a230, type:15, status:2, dag:{ObIDag:{this:0x7f9272929e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67E-0-0, dag_ret:0, dag_status:2, start_time:1727337645650599, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:45.655365] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=5] task finish process(ret=0, start_time=1727337645651504, end_time=1727337645655365, runtime=3861, *this={this:0x7f927292a230, type:15, status:2, dag:{ObIDag:{this:0x7f9272929e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67E-0-0, dag_ret:0, dag_status:2, start_time:1727337645650599, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:45.655385] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=2] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.655546] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=309, range=0x7f92a9a56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:11}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685e6060, count:7, allocator:0x7f926362e7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92685e63b0, local_datums:0x7f92685e63b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:45.655746] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=51][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.655786] ERROR issue_dba_error (ob_log.cpp:1875) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.655789] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.655797] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.655799] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.655801] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:7, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 12, flag: 0, null: 0, ptr: 0x7f92685ea1e8, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea0e0, hex: 420B000000000000, int: 2882},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea118, hex: 0100000000000000, int: 1, num_digit0: 0},idx=4:{len: 16, flag: 0, null: 0, ptr: 0x7f92685ea1f4, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea188, hex: C818154EB7520BE8, int: -1726195084923955000},idx=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea1c0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19292, column_count:11, rowkey_column_count:7, has_column_checksum:0, row_count:156, row_store_type:0, opt:0, var_column_count:0, row_offset:15827, original_length:16391, max_merged_trans_version:1726241316783508000, data_length:16391, data_zlength:16391, data_checksum:3629777942, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6804090, buf_size:16391, data_size:16391, row_count:156, column_count:11, max_merged_trans_version:1726241316783508000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:619385244, row_count_delta:1, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16391}) [2024-09-26 16:00:45.655813] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:45.655815] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:45.655817] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:45.655818] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=1][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.655821] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=0] partition merge iter row count(i=0, row_count=156, ghost_row_count=0, pkey={tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, table={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337645628738, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.655844] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645655505, end_time=1727337645655844, runtime=339, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272929e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67E-0-0, dag_ret:0, dag_status:2, start_time:1727337645650599, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:45.655855] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5256, dag_cnt=34, dag_cnts_[dag.get_type()]=34, &dag=0x7f9272929e70, dag={ObIDag:{this:0x7f9272929e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645650599, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}) [2024-09-26 16:00:45.655864] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67E-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337645650590, task_id:YB42C0A8002D-0006230056D8C67E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.655933] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645655931, task_id:YB42C0A8002D-0006230056D8C683-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.655937] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=3] schedule one task(task=0x7f927292bc20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.655954] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.655959] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.655961] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.655965] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.655968] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.655974] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.656901] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337645628753, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.656918] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9265a7d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.656943] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=20] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.656947] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.656953] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265a7d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:331}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.656972] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:331}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9265a7d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:331}, table_count:1, [{i:0, table_key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}, merge_scn:{val:1726241465393532000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8633ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645652518, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.43ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.656998] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67F-0-0, dag_ret:0, dag_status:2, start_time:1727337645652434, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}, scanned_row_cnt_arr:0x7f9265a7d480, output_block_cnt_arr:0x7f9265a7d488, concurrent_cnt:1, estimate_row_cnt:45, estimate_occupy_size:0, latest_update_ts:1727337645652518, estimated_finish_time:1727337662656998}) [2024-09-26 16:00:45.657005] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f927292a760, type:15, status:2, dag:{ObIDag:{this:0x7f927292a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67F-0-0, dag_ret:0, dag_status:2, start_time:1727337645652434, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:45.657011] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=5] task finish process(ret=0, start_time=1727337645652515, end_time=1727337645657011, runtime=4496, *this={this:0x7f927292a760, type:15, status:2, dag:{ObIDag:{this:0x7f927292a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67F-0-0, dag_ret:0, dag_status:2, start_time:1727337645652434, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:45.657028] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=2] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.657047] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337645628768, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.657058] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753357664, storage_schema:{this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753357664, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.657080] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=331, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264b22060, count:5, allocator:0x7f92a31fe7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9264b22360, local_datums:0x7f9264b22360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:45.657123] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=12] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:45.657127] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.657133] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:458}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753357664, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.657154] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:458}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753357664, storage_schema:{this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:458}, table_count:1, [{i:0, table_key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8627b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645653205, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.93ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.657273] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=20] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:45.657315] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.657356] ERROR issue_dba_error (ob_log.cpp:1875) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.657359] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.657369] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=0] partition merge iter row count(i=0, row_count=196, ghost_row_count=0, pkey={tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, table={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337645628753, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.657393] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645657049, end_time=1727337645657393, runtime=344, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f927292a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67F-0-0, dag_ret:0, dag_status:2, start_time:1727337645652434, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:45.657403] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4969, dag_cnt=33, dag_cnts_[dag.get_type()]=33, &dag=0x7f927292a3a0, dag={ObIDag:{this:0x7f927292a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C67F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645652434, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}) [2024-09-26 16:00:45.657411] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C67F-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645652427, task_id:YB42C0A8002D-0006230056D8C67F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.657424] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C680-0-0, dag_ret:0, dag_status:2, start_time:1727337645653189, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728fb4a0, output_block_cnt_arr:0x7f92728fb4a8, concurrent_cnt:1, estimate_row_cnt:14598, estimate_occupy_size:4194304, latest_update_ts:1727337645653205, estimated_finish_time:1727337662657425}) [2024-09-26 16:00:45.657445] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=21] succeed to init merge ctx(task={this:0x7f927292ac90, type:15, status:2, dag:{ObIDag:{this:0x7f927292a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C680-0-0, dag_ret:0, dag_status:2, start_time:1727337645653189, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.657451] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=6] task finish process(ret=0, start_time=1727337645653203, end_time=1727337645657451, runtime=4248, *this={this:0x7f927292ac90, type:15, status:2, dag:{ObIDag:{this:0x7f927292a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C680-0-0, dag_ret:0, dag_status:2, start_time:1727337645653189, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.657469] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=3] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.657477] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645657475, task_id:YB42C0A8002D-0006230056D8C684-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.657480] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=2] schedule one task(task=0x7f927292c150, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.657514] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.657517] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.657519] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.657524] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.657527] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.657535] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.657588] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=458, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:8, allocator:0x7f92865547d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9263b5e3e0, local_datums:0x7f9263b5e3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:45.657813] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.657848] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.657850] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.657859] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=0] partition merge iter row count(i=0, row_count=247, ghost_row_count=0, pkey={tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337645628768, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.657889] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645657552, end_time=1727337645657889, runtime=337, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f927292a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C680-0-0, dag_ret:0, dag_status:2, start_time:1727337645653189, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.657899] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4710, dag_cnt=32, dag_cnts_[dag.get_type()]=32, &dag=0x7f927292a8d0, dag={ObIDag:{this:0x7f927292a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C680-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645653189, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.657906] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C680-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645653184, task_id:YB42C0A8002D-0006230056D8C680-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.657920] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337645657918, task_id:YB42C0A8002D-0006230056D8C685-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.657923] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=3] schedule one task(task=0x7f927292c680, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.657949] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.657952] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.657954] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.657958] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.657961] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b862a0a0, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:490, tablet_addr:[816017-1849-0-2305843629838340096-2](offset=1871872,size=578,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:472}, data_tablet_id:{id:472}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:19976489, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862a734, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844179594129408-0](offset=1847296,size=1090,type=2,seq=0), type:"MAJOR", tablet_id:{id:472}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389523010000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843910084932674-0](offset=1848386,size=839,type=2,seq=0), type:"MINOR", tablet_id:{id:472}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178194887823000, upper_trans_version:1726178194887823000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337645628847, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd4080, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209505593}, this:0x7f92a3bd4080, timestamp:1727334209505593, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:65, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274254072414000, v:0}, rec_scn:{val:1726241829658678000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697839104-2](offset=1859584,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834376524-0](offset=1859916,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881206236-0](offset=1860060,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447726270-0](offset=1860286,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7ac60, 0x7f92a3bd4080, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.658001] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=39] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.658007] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.658047] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337645628799, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.658056] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753332736, storage_schema:{this:0x7f9265a81330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=36, max_schema_version_in_memtable=1710828753332736, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.658109] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337645628783, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.658121] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753417144, storage_schema:{this:0x7f9265a7f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828753417144, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.658195] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:45.658199] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.658205] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:45.658208] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.658205] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265a81330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:455}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:10, rowkey_column_count:9, schema_rowkey_col_cnt:7, full_stored_col_cnt:38, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753332736, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=26 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.658214] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265a7f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:487}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753417144, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.658226] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:455}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753332736, storage_schema:{this:0x7f9265a81330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:455}, table_count:1, [{i:0, table_key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645654003, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.20ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.658232] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:487}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753417144, storage_schema:{this:0x7f9265a7f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:487}, table_count:1, [{i:0, table_key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645653678, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.53ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.658342] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=26] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:45.658387] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=29] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:45.658452] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C681-0-0, dag_ret:0, dag_status:2, start_time:1727337645653658, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265a7f4a0, output_block_cnt_arr:0x7f9265a7f4a8, concurrent_cnt:1, estimate_row_cnt:2199, estimate_occupy_size:0, latest_update_ts:1727337645653678, estimated_finish_time:1727337662658451}) [2024-09-26 16:00:45.658460] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f927292b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f927292ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C681-0-0, dag_ret:0, dag_status:2, start_time:1727337645653658, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.658465] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=5] task finish process(ret=0, start_time=1727337645653676, end_time=1727337645658465, runtime=4789, *this={this:0x7f927292b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f927292ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C681-0-0, dag_ret:0, dag_status:2, start_time:1727337645653658, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.658477] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=2] schedule one task(task=0x7f9264b46030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.658581] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C682-0-0, dag_ret:0, dag_status:2, start_time:1727337645653952, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265a814b0, output_block_cnt_arr:0x7f9265a814b8, concurrent_cnt:1, estimate_row_cnt:3824, estimate_occupy_size:4194304, latest_update_ts:1727337645654003, estimated_finish_time:1727337662658583}) [2024-09-26 16:00:45.658592] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=10] succeed to init merge ctx(task={this:0x7f927292b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f927292b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C682-0-0, dag_ret:0, dag_status:2, start_time:1727337645653952, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.658597] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=4] task finish process(ret=0, start_time=1727337645654002, end_time=1727337645658597, runtime=4595, *this={this:0x7f927292b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f927292b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C682-0-0, dag_ret:0, dag_status:2, start_time:1727337645653952, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.658616] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=6] schedule one task(task=0x7f9264b36030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.658652] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=487, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:11}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:8, allocator:0x7f92b33fe7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9263a383e0, local_datums:0x7f9263a383e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:45.658676] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=455, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:36, schema_rowkey_cnt:7, rowkey_cnt:9, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:7, column_cnt:38}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:9, allocator:0x7f92864107d8, capacity:9, init_cnt:9}, datum_utils:{is_oracle_mode:false, rowkey_cnt:9, col_cnt:9, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:38, datum_buffer:{capacity:38, datums:0x7f92684b5340, local_datums:0x7f92684b43f0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,]}) [2024-09-26 16:00:45.658876] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.658913] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.658916] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.658933] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.658926] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=0] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337645628783, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.658945] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645658620, end_time=1727337645658945, runtime=325, *this={this:0x7f9264b46030, type:1, status:2, dag:{ObIDag:{this:0x7f927292ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C681-0-0, dag_ret:0, dag_status:2, start_time:1727337645653658, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.658954] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5296, dag_cnt=31, dag_cnts_[dag.get_type()]=31, &dag=0x7f927292ae00, dag={ObIDag:{this:0x7f927292ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C681-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645653658, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.658962] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C681-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337645653653, task_id:YB42C0A8002D-0006230056D8C681-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.658971] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.658974] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.658987] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645658984, task_id:YB42C0A8002D-0006230056D8C686-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.658991] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=4] schedule one task(task=0x7f927292cbb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.658984] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=0] partition merge iter row count(i=0, row_count=140, ghost_row_count=0, pkey={tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337645628799, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.659004] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645658636, end_time=1727337645659004, runtime=368, *this={this:0x7f9264b36030, type:1, status:2, dag:{ObIDag:{this:0x7f927292b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C682-0-0, dag_ret:0, dag_status:2, start_time:1727337645653952, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.659012] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337645627869, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.659023] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=10] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:45.659025] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5073, dag_cnt=30, dag_cnts_[dag.get_type()]=30, &dag=0x7f927292b330, dag={ObIDag:{this:0x7f927292b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C682-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645653952, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.659026] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9268515480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.659033] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C682-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337645653949, task_id:YB42C0A8002D-0006230056D8C682-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.659047] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645659046, task_id:YB42C0A8002D-0006230056D8C687-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.659050] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=2] schedule one task(task=0x7f927292d0e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.659156] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3214, total_rows=22498) [2024-09-26 16:00:45.659160] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.659166] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847408}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.659182] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:1152921504606847408}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9268515480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847408}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645648287, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=10.88ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.659295] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272927f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C678-0-0, dag_ret:0, dag_status:2, start_time:1727337645647371, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685155d0, output_block_cnt_arr:0x7f92685155d8, concurrent_cnt:1, estimate_row_cnt:35098, estimate_occupy_size:18874368, latest_update_ts:1727337645648287, estimated_finish_time:1727337662659304}) [2024-09-26 16:00:45.659304] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9272928310, type:15, status:2, dag:{ObIDag:{this:0x7f9272927f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C678-0-0, dag_ret:0, dag_status:2, start_time:1727337645647371, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.659309] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=5] task finish process(ret=0, start_time=1727337645648284, end_time=1727337645659309, runtime=11025, *this={this:0x7f9272928310, type:15, status:2, dag:{ObIDag:{this:0x7f9272927f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C678-0-0, dag_ret:0, dag_status:2, start_time:1727337645647371, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.659327] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=1] schedule one task(task=0x7f9272934ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.659339] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.659342] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.659344] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726077706339704027) [2024-09-26 16:00:45.659348] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.659351] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.659357] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.659377] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.659379] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.659381] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.659384] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.659382] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=1152921504606847408, range=0x7f92aad52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:5, allocator:0x7f92b33d87d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5a360, local_datums:0x7f9263b5a360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:45.659386] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8622d40, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:492, tablet_addr:[816017-1849-0-2305843631985766400-2](offset=1814528,size=580,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:140}, data_tablet_id:{id:140}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726073917016589000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2305242685, row_count:150}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86233d4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844315959259136-0](offset=1765376,size=1217,type=2,seq=0), type:"MAJOR", tablet_id:{id:140}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726070403667750001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843943370847425-0](offset=1766593,size=870,type=2,seq=0), type:"MINOR", tablet_id:{id:140}, scn_range:{start_scn:{val:1726073917016589000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726167395874925000, upper_trans_version:1726167395874925000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337645628880, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcfc00, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495712}, this:0x7f92a3bcfc00, timestamp:1727334209495712, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:57, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726256203821911000, v:0}, rec_scn:{val:1726241803398655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697781760-2](offset=1802240,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834319180-0](offset=1802572,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881148892-0](offset=1802716,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447668926-0](offset=1802942,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7b430, 0x7f92a3bcfc00, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.659424] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=38] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.659428] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.659670] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.659728] ERROR issue_dba_error (ob_log.cpp:1875) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=17][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.659731] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.659740] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=0] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337645627869, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.659762] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645659346, end_time=1727337645659762, runtime=416, *this={this:0x7f9272934ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9272927f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C678-0-0, dag_ret:0, dag_status:2, start_time:1727337645647371, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.659772] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=12401, dag_cnt=29, dag_cnts_[dag.get_type()]=29, &dag=0x7f9272927f50, dag={ObIDag:{this:0x7f9272927f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C678-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645647371, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.659780] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C678-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337645647365, task_id:YB42C0A8002D-0006230056D8C678-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.659806] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645659805, task_id:YB42C0A8002D-0006230056D8C688-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.659810] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=3] schedule one task(task=0x7f927292d610, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.660615] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.660620] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.660623] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.660627] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.660631] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.660637] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.660671] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:45.660698] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337645628831, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.660704] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337645628814, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.660709] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753441792, storage_schema:{this:0x7f92728f9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828753441792, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.660716] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753437040, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=57, max_schema_version_in_memtable=1710828753437040, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.660764] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1295, total_rows=1295) [2024-09-26 16:00:45.660767] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.660775] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1362, total_rows=1362) [2024-09-26 16:00:45.660778] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.660772] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:501}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753441792, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.660784] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:492}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:12, rowkey_column_count:11, schema_rowkey_col_cnt:9, full_stored_col_cnt:59, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753437040, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=28 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.660790] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:501}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753441792, storage_schema:{this:0x7f92728f9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:501}, table_count:1, [{i:0, table_key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645657508, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.26ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.660808] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=19] succeed to build merge ctx(tablet_id={id:492}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753437040, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:492}, table_count:1, [{i:0, table_key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8624c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645655949, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.83ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.660859] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C684-0-0, dag_ret:0, dag_status:2, start_time:1727337645657479, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f9490, output_block_cnt_arr:0x7f92728f9498, concurrent_cnt:1, estimate_row_cnt:1295, estimate_occupy_size:0, latest_update_ts:1727337645657508, estimated_finish_time:1727337662660859}) [2024-09-26 16:00:45.660865] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292c150, type:15, status:2, dag:{ObIDag:{this:0x7f927292bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C684-0-0, dag_ret:0, dag_status:2, start_time:1727337645657479, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.660870] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=5] task finish process(ret=0, start_time=1727337645657506, end_time=1727337645660870, runtime=3364, *this={this:0x7f927292c150, type:15, status:2, dag:{ObIDag:{this:0x7f927292bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C684-0-0, dag_ret:0, dag_status:2, start_time:1727337645657479, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.660885] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C683-0-0, dag_ret:0, dag_status:2, start_time:1727337645655937, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab4c0, output_block_cnt_arr:0x7f92684ab4c8, concurrent_cnt:1, estimate_row_cnt:1362, estimate_occupy_size:2097152, latest_update_ts:1727337645655949, estimated_finish_time:1727337662660886}) [2024-09-26 16:00:45.660892] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f927292bc20, type:15, status:2, dag:{ObIDag:{this:0x7f927292b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C683-0-0, dag_ret:0, dag_status:2, start_time:1727337645655937, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.660897] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=5] task finish process(ret=0, start_time=1727337645655947, end_time=1727337645660897, runtime=4950, *this={this:0x7f927292bc20, type:15, status:2, dag:{ObIDag:{this:0x7f927292b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C683-0-0, dag_ret:0, dag_status:2, start_time:1727337645655937, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.660891] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337645628847, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.660913] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=22] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753360568, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753360568, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.660919] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=1] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.660925] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=3] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.660922] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.660934] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=11] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.660940] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:472}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753360568, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.660954] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:472}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753360568, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:472}, table_count:1, [{i:0, table_key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645657944, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.99ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.660972] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C685-0-0, dag_ret:0, dag_status:2, start_time:1727337645657923, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5460, output_block_cnt_arr:0x7f9287db5468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337645657944, estimated_finish_time:1727337662660972}) [2024-09-26 16:00:45.660978] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f927292c680, type:15, status:2, dag:{ObIDag:{this:0x7f927292c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C685-0-0, dag_ret:0, dag_status:2, start_time:1727337645657923, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.660992] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=14] task finish process(ret=0, start_time=1727337645657942, end_time=1727337645660992, runtime=3050, *this={this:0x7f927292c680, type:15, status:2, dag:{ObIDag:{this:0x7f927292c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C685-0-0, dag_ret:0, dag_status:2, start_time:1727337645657923, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.661015] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=2] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.661057] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=501, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:6, allocator:0x7f92865b67d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92665f8390, local_datums:0x7f92665f8390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:45.661153] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.661198] ERROR issue_dba_error (ob_log.cpp:1875) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.661202] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.661212] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=0] partition merge iter row count(i=0, row_count=56, ghost_row_count=0, pkey={tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337645628831, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.661233] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645661019, end_time=1727337645661233, runtime=214, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f927292bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C684-0-0, dag_ret:0, dag_status:2, start_time:1727337645657479, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.661242] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3763, dag_cnt=28, dag_cnts_[dag.get_type()]=28, &dag=0x7f927292bd90, dag={ObIDag:{this:0x7f927292bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C684-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645657479, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.661250] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C684-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645657475, task_id:YB42C0A8002D-0006230056D8C684-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.661273] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645661272, task_id:YB42C0A8002D-0006230056D8C689-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.661277] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=3] schedule one task(task=0x7f927292db40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.661313] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=492, range=0x7f92a9d52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:57, schema_rowkey_cnt:9, rowkey_cnt:11, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:9, column_cnt:59}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92728f8060, count:11, allocator:0x7f9269f787d8, capacity:11, init_cnt:11}, datum_utils:{is_oracle_mode:false, rowkey_cnt:11, col_cnt:11, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:59, datum_buffer:{capacity:59, datums:0x7f9263a3e050, local_datums:0x7f92728f8440}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,col_id=43:NOP,col_id=44:NOP,col_id=45:NOP,col_id=46:NOP,col_id=47:NOP,col_id=48:NOP,col_id=49:NOP,col_id=50:NOP,col_id=51:NOP,col_id=52:NOP,col_id=53:NOP,col_id=54:NOP,col_id=55:NOP,col_id=56:NOP,col_id=57:NOP,col_id=58:NOP,]}) [2024-09-26 16:00:45.661528] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=18][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.661560] ERROR issue_dba_error (ob_log.cpp:1875) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.661563] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.661571] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=0] partition merge iter row count(i=0, row_count=61, ghost_row_count=0, pkey={tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337645628814, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.661589] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645661276, end_time=1727337645661589, runtime=313, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f927292b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C683-0-0, dag_ret:0, dag_status:2, start_time:1727337645655937, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.661597] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5660, dag_cnt=27, dag_cnts_[dag.get_type()]=27, &dag=0x7f927292b860, dag={ObIDag:{this:0x7f927292b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C683-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645655937, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.661604] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C683-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645655931, task_id:YB42C0A8002D-0006230056D8C683-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.661640] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645661638, task_id:YB42C0A8002D-0006230056D8C68A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.661643] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=2] schedule one task(task=0x7f927292e070, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.661665] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.661668] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.661670] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:45.661675] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.661679] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.661686] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.661699] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=472, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ee060, count:3, allocator:0x7f92636307d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f92664ee320, local_datums:0x7f92664ee320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:45.661754] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=31][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.661787] ERROR issue_dba_error (ob_log.cpp:1875) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.661790] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.661797] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=0] partition merge iter row count(i=0, row_count=17, ghost_row_count=0, pkey={tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337645628847, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.661815] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645661667, end_time=1727337645661815, runtime=148, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f927292c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C685-0-0, dag_ret:0, dag_status:2, start_time:1727337645657923, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.661824] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3901, dag_cnt=26, dag_cnts_[dag.get_type()]=26, &dag=0x7f927292c2c0, dag={ObIDag:{this:0x7f927292c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C685-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645657923, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.661830] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C685-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337645657918, task_id:YB42C0A8002D-0006230056D8C685-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.661861] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.661864] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.661866] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:45.661871] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.661870] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337645661868, task_id:YB42C0A8002D-0006230056D8C68B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.661875] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=5] schedule one task(task=0x7f927292e5a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.661890] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337645628925, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.661897] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.661901] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.661904] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:45.661900] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752949376, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=20, max_schema_version_in_memtable=1710828752949376, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.661908] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.661912] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.661918] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.661921] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.661925] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.661930] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:323}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:22, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752949376, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.661875] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8642d40, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:487, tablet_addr:[816017-1849-0-2305843626617085952-2](offset=1843200,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101010}, data_tablet_id:{id:140}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726073917016589000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4294280206, row_count:150}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86433f8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844211806371840-0](offset=1835008,size=1120,type=2,seq=0), type:"MAJOR", tablet_id:{id:101010}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726070403667750001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843968066978912-0](offset=1836128,size=893,type=2,seq=0), type:"MINOR", tablet_id:{id:101010}, scn_range:{start_scn:{val:1726073917016589000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726167395874925000, upper_trans_version:1726167395874925000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337645628910, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd1370, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495895}, this:0x7f92a3bd1370, timestamp:1727334209495895, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:60, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726256203821911000, v:0}, rec_scn:{val:1726241803398655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697818624-2](offset=1839104,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276806988-0](offset=1839436,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881185762-0](offset=1839586,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7c3d0, 0x7f92a3bd1370, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.661962] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.661948] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:323}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752949376, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:323}, table_count:1, [{i:0, table_key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645661658, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=270us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.661969] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.661977] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68A-0-0, dag_ret:0, dag_status:2, start_time:1727337645661642, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505480, output_block_cnt_arr:0x7f9266505488, concurrent_cnt:1, estimate_row_cnt:205, estimate_occupy_size:0, latest_update_ts:1727337645661658, estimated_finish_time:1727337662661977}) [2024-09-26 16:00:45.661984] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292e070, type:15, status:2, dag:{ObIDag:{this:0x7f927292dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68A-0-0, dag_ret:0, dag_status:2, start_time:1727337645661642, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.661989] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=4] task finish process(ret=0, start_time=1727337645661655, end_time=1727337645661988, runtime=333, *this={this:0x7f927292e070, type:15, status:2, dag:{ObIDag:{this:0x7f927292dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68A-0-0, dag_ret:0, dag_status:2, start_time:1727337645661642, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662003] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=3] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.662042] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=323, range=0x7f92a9cd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:20, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:22}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fc060, count:5, allocator:0x7f92a31fe7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f92665fc360, local_datums:0x7f92665fc360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,]}) [2024-09-26 16:00:45.662056] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337645628864, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.662064] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753001832, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=21, max_schema_version_in_memtable=1710828753001832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.662076] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.662079] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.662084] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:342}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:23, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753001832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.662101] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:342}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753001832, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:342}, table_count:1, [{i:0, table_key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645659333, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.75ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.662123] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C686-0-0, dag_ret:0, dag_status:2, start_time:1727337645658991, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db7490, output_block_cnt_arr:0x7f9287db7498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337645659333, estimated_finish_time:1727337662662122}) [2024-09-26 16:00:45.662128] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f927292cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f927292c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C686-0-0, dag_ret:0, dag_status:2, start_time:1727337645658991, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662154] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=26] task finish process(ret=0, start_time=1727337645659332, end_time=1727337645662154, runtime=2822, *this={this:0x7f927292cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f927292c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C686-0-0, dag_ret:0, dag_status:2, start_time:1727337645658991, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662189] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337645628948, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.662200] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=41, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.662201] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337645628895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.662211] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.662230] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=29] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.662236] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=24] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.662239] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.662242] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.662246] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101009}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:45.662243] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:377}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:43, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.662261] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:377}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:377}, table_count:1, [{i:0, table_key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645661891, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=349us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.662261] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:101009}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101009}, table_count:1, [{i:0, table_key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8641da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645660610, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.63ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.662282] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68B-0-0, dag_ret:0, dag_status:2, start_time:1727337645661875, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92664ed480, output_block_cnt_arr:0x7f92664ed488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337645661891, estimated_finish_time:1727337662662282}) [2024-09-26 16:00:45.662284] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C688-0-0, dag_ret:0, dag_status:2, start_time:1727337645659809, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad470, output_block_cnt_arr:0x7f92684ad478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337645660610, estimated_finish_time:1727337662662284}) [2024-09-26 16:00:45.662288] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f927292e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68B-0-0, dag_ret:0, dag_status:2, start_time:1727337645661875, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662289] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f927292d610, type:15, status:2, dag:{ObIDag:{this:0x7f927292d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C688-0-0, dag_ret:0, dag_status:2, start_time:1727337645659809, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662293] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=4] task finish process(ret=0, start_time=1727337645661889, end_time=1727337645662293, runtime=404, *this={this:0x7f927292e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f927292e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68B-0-0, dag_ret:0, dag_status:2, start_time:1727337645661875, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662294] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=4] task finish process(ret=0, start_time=1727337645660607, end_time=1727337645662293, runtime=1686, *this={this:0x7f927292d610, type:15, status:2, dag:{ObIDag:{this:0x7f927292d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C688-0-0, dag_ret:0, dag_status:2, start_time:1727337645659809, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662322] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337645628880, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.662331] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9263b59330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.662339] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=5] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.662341] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.662348] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.662345] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263b59330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:140}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.662358] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:140}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9263b59330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:140}, table_count:1, [{i:0, table_key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8622d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645659374, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.97ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.662376] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=12] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C687-0-0, dag_ret:0, dag_status:2, start_time:1727337645659050, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263b59460, output_block_cnt_arr:0x7f9263b59468, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337645659374, estimated_finish_time:1727337662662376}) [2024-09-26 16:00:45.662381] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f927292d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f927292cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C687-0-0, dag_ret:0, dag_status:2, start_time:1727337645659050, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662385] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=4] task finish process(ret=0, start_time=1727337645659373, end_time=1727337645662385, runtime=3012, *this={this:0x7f927292d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f927292cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C687-0-0, dag_ret:0, dag_status:2, start_time:1727337645659050, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662389] ERROR issue_dba_error (ob_log.cpp:1875) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.662392] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.662399] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337645628910, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.662406] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.662402] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=0] partition merge iter row count(i=0, row_count=171, ghost_row_count=0, pkey={tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337645628925, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.662413] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=5] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.662415] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.662419] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101010}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:45.662424] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645662014, end_time=1727337645662424, runtime=410, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f927292dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68A-0-0, dag_ret:0, dag_status:2, start_time:1727337645661642, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662434] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=791, dag_cnt=25, dag_cnts_[dag.get_type()]=25, &dag=0x7f927292dcb0, dag={ObIDag:{this:0x7f927292dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645661642, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.662430] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=8] succeed to build merge ctx(tablet_id={id:101010}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101010}, table_count:1, [{i:0, table_key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8642d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645661855, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=563us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.662441] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68A-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645661638, task_id:YB42C0A8002D-0006230056D8C68A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.662446] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=12] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C689-0-0, dag_ret:0, dag_status:2, start_time:1727337645661277, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab470, output_block_cnt_arr:0x7f92684ab478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337645661855, estimated_finish_time:1727337662662445}) [2024-09-26 16:00:45.662465] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=19] succeed to init merge ctx(task={this:0x7f927292db40, type:15, status:2, dag:{ObIDag:{this:0x7f927292d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C689-0-0, dag_ret:0, dag_status:2, start_time:1727337645661277, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662470] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=5] task finish process(ret=0, start_time=1727337645661854, end_time=1727337645662470, runtime=616, *this={this:0x7f927292db40, type:15, status:2, dag:{ObIDag:{this:0x7f927292d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C689-0-0, dag_ret:0, dag_status:2, start_time:1727337645661277, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662474] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=3] schedule one task(task=0x7f9272934ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.662479] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=4] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.662483] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=1] schedule one task(task=0x7f9272935260, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.662488] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=2] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.662494] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645662492, task_id:YB42C0A8002D-0006230056D8C68C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.662497] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=2] schedule one task(task=0x7f927292ead0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.662524] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.662528] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.662530] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:45.662535] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.662531] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=342, range=0x7f92ab152200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:21, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:23}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d36060, count:6, allocator:0x7f9269ffe7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:23, datum_buffer:{capacity:32, datums:0x7f9287d36390, local_datums:0x7f9287d36390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,]}) [2024-09-26 16:00:45.662562] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.662538] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864eec0, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[814042-5586-0-2305843650238771200-2](offset=1208320,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:297}, data_tablet_id:{id:297}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726063409221604000, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2361333062, row_count:24}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844531780800512-0](offset=1200128,size=1418,type=2,seq=0), type:"MAJOR", tablet_id:{id:297}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063202518238000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843919747962250-0](offset=1201546,size=848,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726063409221604000, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557514739000, upper_trans_version:1726121557514739000}, {i:1, addr:[814042-5586-0-2305843920821704922-0](offset=1202394,size=849,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149603136610000, upper_trans_version:1726149603136610000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337645628966, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd86d0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220478269}, this:0x7f92a3bd86d0, timestamp:1727334220478269, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:74, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726264804632655000, v:0}, rec_scn:{val:1726264804632655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513817088-2](offset=1204224,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843183161074102-0](offset=1204662,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880551000-0](offset=1204824,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447071034-0](offset=1205050,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7eae0, 0x7f92a3bd86d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.662567] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101009, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dd0060, count:4, allocator:0x7f92636807d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287dd0350, local_datums:0x7f9287dd0350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:45.662578] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=40] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.662584] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.662596] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=1] schedule one task(task=0x7f9262cbe030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.662608] ERROR issue_dba_error (ob_log.cpp:1875) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.662609] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.662611] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.662618] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.662621] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.662622] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c0a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 12, flag: 0, null: 0, ptr: 0x7f9265b0c1b0, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c118, hex: 420B000000000000, int: 2882},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c150, hex: 98A4BFDF9C5C0BE8, int: -1726184203327921000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c188, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:18768, column_count:23, rowkey_column_count:6, has_column_checksum:0, row_count:3, row_store_type:0, opt:4, var_column_count:0, row_offset:494, original_length:446, max_merged_trans_version:1726184204644606000, data_length:446, data_zlength:446, data_checksum:3585221911, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92b6e04090, buf_size:446, data_size:446, row_count:3, column_count:23, max_merged_trans_version:1726184204644606000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1373226696, row_count_delta:1, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:446}) [2024-09-26 16:00:45.662635] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=13][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:45.662637] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=1][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:00:45.662638] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=1][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:00:45.662640] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=2][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:00:45.662642] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=2][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:00:45.662645] ERROR issue_dba_error (ob_log.cpp:1875) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.662647] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.662644] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=1] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337645628864, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.662657] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=10][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:45.662656] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337645628895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.662665] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:45.662668] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:45.662670] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337645662487, end_time=1727337645662669, runtime=182, *this={this:0x7f9272934ed0, type:1, status:2, dag:{ObIDag:{this:0x7f927292c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C686-0-0, dag_ret:0, dag_status:2, start_time:1727337645658991, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662675] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=5][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9272934ed0, type:1, status:2, dag:{ObIDag:{this:0x7f927292c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C686-0-0, dag_ret:0, dag_status:2, start_time:1727337645658991, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.662676] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645662527, end_time=1727337645662676, runtime=149, *this={this:0x7f9272935260, type:1, status:2, dag:{ObIDag:{this:0x7f927292d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C688-0-0, dag_ret:0, dag_status:2, start_time:1727337645659809, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662682] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=3691, dag_cnt=24, dag_cnts_[dag.get_type()]=24, &dag=0x7f927292c7f0, dag={ObIDag:{this:0x7f927292c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C686-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645658991, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.662689] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C686-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645658984, task_id:YB42C0A8002D-0006230056D8C686-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.662705] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645662704, task_id:YB42C0A8002D-0006230056D8C68D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.662709] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=3] schedule one task(task=0x7f927292f000, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.662723] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.662725] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.662727] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:45.662729] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.662731] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.662735] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.662729] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101010, range=0x7f92aaf52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665ac060, count:4, allocator:0x7f92b32387d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665ac350, local_datums:0x7f92665ac350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:45.662760] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.662769] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=377, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:41, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:43}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b02060, count:5, allocator:0x7f92636287d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:43, datum_buffer:{capacity:43, datums:0x7f9265b032b0, local_datums:0x7f9265b02360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,]}) [2024-09-26 16:00:45.662788] ERROR issue_dba_error (ob_log.cpp:1875) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.662790] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.662797] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337645628910, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.662809] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.662813] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645662690, end_time=1727337645662812, runtime=122, *this={this:0x7f9262cbe030, type:1, status:2, dag:{ObIDag:{this:0x7f927292d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C689-0-0, dag_ret:0, dag_status:2, start_time:1727337645661277, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662820] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1543, dag_cnt=23, dag_cnts_[dag.get_type()]=23, &dag=0x7f927292d780, dag={ObIDag:{this:0x7f927292d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C689-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645661277, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.662827] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C689-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645661272, task_id:YB42C0A8002D-0006230056D8C689-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.662839] ERROR issue_dba_error (ob_log.cpp:1875) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.662839] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337645662838, task_id:YB42C0A8002D-0006230056D8C68E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.662842] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=2] schedule one task(task=0x7f927292f530, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.662841] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.662850] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=0] partition merge iter row count(i=0, row_count=6, ghost_row_count=0, pkey={tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337645628948, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.662867] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645662732, end_time=1727337645662867, runtime=135, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f927292e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68B-0-0, dag_ret:0, dag_status:2, start_time:1727337645661875, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.662876] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1001, dag_cnt=22, dag_cnts_[dag.get_type()]=22, &dag=0x7f927292e1e0, dag={ObIDag:{this:0x7f927292e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645661875, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.662883] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645661868, task_id:YB42C0A8002D-0006230056D8C68B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.662886] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=140, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264b1c060, count:3, allocator:0x7f926367e7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264b1c320, local_datums:0x7f9264b1c320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:45.662895] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645662894, task_id:YB42C0A8002D-0006230056D8C68F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.662898] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=2] schedule one task(task=0x7f927292fa60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.662945] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.662949] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.662951] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:45.662955] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.662958] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.662964] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.663005] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.663041] ERROR issue_dba_error (ob_log.cpp:1875) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.663044] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.663052] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337645628880, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.663086] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645662858, end_time=1727337645663086, runtime=228, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f927292cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C687-0-0, dag_ret:0, dag_status:2, start_time:1727337645659050, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.663096] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4046, dag_cnt=21, dag_cnts_[dag.get_type()]=21, &dag=0x7f927292cd20, dag={ObIDag:{this:0x7f927292cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C687-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645659050, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.663104] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C687-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645659046, task_id:YB42C0A8002D-0006230056D8C687-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.663137] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337645663133, task_id:YB42C0A8002D-0006230056D8C690-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.663143] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=6] schedule one task(task=0x7f927292ff90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.663163] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.663166] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.663168] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:45.663172] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.663175] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.663180] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.663183] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.663187] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.663189] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:45.663193] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.663197] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.663203] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.663171] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337645628981, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.663208] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=37] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9265b0b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.663224] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.663227] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3418, dag_cnt=20, dag_cnts_[dag.get_type()]=20, &dag=0x7f927292d250, dag={ObIDag:{this:0x7f927292d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C688-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645659809, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.663236] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C688-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337645659805, task_id:YB42C0A8002D-0006230056D8C688-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.663233] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337645628966, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.663233] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337645629012, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.663244] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.663241] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=16, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.663256] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.663259] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.663261] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=15] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.663264] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.663264] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101066}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.663270] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:297}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:18, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.663282] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:101066}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101066}, table_count:1, [{i:0, table_key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8653ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645662940, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=322us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.663289] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:297}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:297}, table_count:1, [{i:0, table_key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645662511, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=757us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.663309] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68F-0-0, dag_ret:0, dag_status:2, start_time:1727337645662898, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505490, output_block_cnt_arr:0x7f9266505498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337645662940, estimated_finish_time:1727337662663309}) [2024-09-26 16:00:45.663312] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68C-0-0, dag_ret:0, dag_status:2, start_time:1727337645662496, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a43490, output_block_cnt_arr:0x7f9263a43498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337645662511, estimated_finish_time:1727337662663311}) [2024-09-26 16:00:45.663315] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292fa60, type:15, status:2, dag:{ObIDag:{this:0x7f927292f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68F-0-0, dag_ret:0, dag_status:2, start_time:1727337645662898, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.663319] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292ead0, type:15, status:2, dag:{ObIDag:{this:0x7f927292e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68C-0-0, dag_ret:0, dag_status:2, start_time:1727337645662496, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.663320] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=4] task finish process(ret=0, start_time=1727337645662938, end_time=1727337645663320, runtime=382, *this={this:0x7f927292fa60, type:15, status:2, dag:{ObIDag:{this:0x7f927292f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68F-0-0, dag_ret:0, dag_status:2, start_time:1727337645662898, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.663324] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=4] task finish process(ret=0, start_time=1727337645662509, end_time=1727337645663323, runtime=814, *this={this:0x7f927292ead0, type:15, status:2, dag:{ObIDag:{this:0x7f927292e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68C-0-0, dag_ret:0, dag_status:2, start_time:1727337645662496, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.663355] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.663360] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265b0b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101065}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.663367] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337645629027, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.663375] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=19, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.663374] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:101065}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9265b0b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101065}, table_count:1, [{i:0, table_key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645662719, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=640us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.663392] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68D-0-0, dag_ret:0, dag_status:2, start_time:1727337645662708, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265b0b490, output_block_cnt_arr:0x7f9265b0b498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337645662719, estimated_finish_time:1727337662663391}) [2024-09-26 16:00:45.663396] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=4] succeed to init merge ctx(task={this:0x7f927292f000, type:15, status:2, dag:{ObIDag:{this:0x7f927292ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68D-0-0, dag_ret:0, dag_status:2, start_time:1727337645662708, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.663400] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=4] task finish process(ret=0, start_time=1727337645662718, end_time=1727337645663400, runtime=682, *this={this:0x7f927292f000, type:15, status:2, dag:{ObIDag:{this:0x7f927292ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68D-0-0, dag_ret:0, dag_status:2, start_time:1727337645662708, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.663413] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.663413] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=4] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.663419] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=3] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.663450] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=28] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.663474] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645663472, task_id:YB42C0A8002D-0006230056D8C691-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.663478] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=4] schedule one task(task=0x7f92729304c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.663501] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=297, range=0x7f92aaad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:16, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:18}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b56060, count:6, allocator:0x7f92c1d9e7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:18, datum_buffer:{capacity:32, datums:0x7f9263b56390, local_datums:0x7f9263b56390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,]}) [2024-09-26 16:00:45.663530] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.663565] ERROR issue_dba_error (ob_log.cpp:1875) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.663567] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.663576] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337645628966, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.663595] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645663434, end_time=1727337645663595, runtime=161, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f927292e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68C-0-0, dag_ret:0, dag_status:2, start_time:1727337645662496, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.663604] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1108, dag_cnt=19, dag_cnts_[dag.get_type()]=19, &dag=0x7f927292e710, dag={ObIDag:{this:0x7f927292e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645662496, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.663611] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68C-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645662492, task_id:YB42C0A8002D-0006230056D8C68C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.663647] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645663646, task_id:YB42C0A8002D-0006230056D8C692-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.663651] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=3] schedule one task(task=0x7f92729309f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.663661] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101066, range=0x7f92a9b52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:7}, cols_desc:{ObIArray:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b04060, count:7, allocator:0x7f92b32387d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9265b043b0, local_datums:0x7f9265b043b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:45.663789] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.663817] ERROR issue_dba_error (ob_log.cpp:1875) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.663820] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.663827] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=0] partition merge iter row count(i=0, row_count=182, ghost_row_count=0, pkey={tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337645629012, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.663843] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645663636, end_time=1727337645663843, runtime=207, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f927292f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68F-0-0, dag_ret:0, dag_status:2, start_time:1727337645662898, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.663851] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=1] dag finished(dag_ret=-4184, runtime=953, dag_cnt=18, dag_cnts_[dag.get_type()]=18, &dag=0x7f927292f6a0, dag={ObIDag:{this:0x7f927292f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645662898, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.663857] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68F-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645662894, task_id:YB42C0A8002D-0006230056D8C68F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.663884] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645663883, task_id:YB42C0A8002D-0006230056D8C693-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.663887] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=3] schedule one task(task=0x7f9272930f20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.663919] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101065, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685ea060, count:6, allocator:0x7f92b32407d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92685ea390, local_datums:0x7f92685ea390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:45.663946] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.663975] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.663977] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.663983] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=0] partition merge iter row count(i=0, row_count=15, ghost_row_count=0, pkey={tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337645628981, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.663998] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645663892, end_time=1727337645663998, runtime=106, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f927292ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68D-0-0, dag_ret:0, dag_status:2, start_time:1727337645662708, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.664005] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1297, dag_cnt=17, dag_cnts_[dag.get_type()]=17, &dag=0x7f927292ec40, dag={ObIDag:{this:0x7f927292ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645662708, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.664011] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68D-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645662704, task_id:YB42C0A8002D-0006230056D8C68D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.664022] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645664021, task_id:YB42C0A8002D-0006230056D8C694-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.664024] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=2] schedule one task(task=0x7f9272931450, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.664031] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.664038] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:334}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:21, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.664058] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:334}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:334}, table_count:1, [{i:0, table_key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8654c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645663159, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=877us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.664125] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C690-0-0, dag_ret:0, dag_status:2, start_time:1727337645663143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a454a0, output_block_cnt_arr:0x7f9263a454a8, concurrent_cnt:1, estimate_row_cnt:993, estimate_occupy_size:0, latest_update_ts:1727337645663159, estimated_finish_time:1727337662664125}) [2024-09-26 16:00:45.664133] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f927292ff90, type:15, status:2, dag:{ObIDag:{this:0x7f927292fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C690-0-0, dag_ret:0, dag_status:2, start_time:1727337645663143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.664138] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=5] task finish process(ret=0, start_time=1727337645663157, end_time=1727337645664138, runtime=981, *this={this:0x7f927292ff90, type:15, status:2, dag:{ObIDag:{this:0x7f927292fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C690-0-0, dag_ret:0, dag_status:2, start_time:1727337645663143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.664146] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=2] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.665841] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.665846] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.665849] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:45.665853] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.665857] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.665864] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.665921] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=334, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:19, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:21}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a60060, count:8, allocator:0x7f92864e07d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:21, datum_buffer:{capacity:32, datums:0x7f9264a603e0, local_datums:0x7f9264a603e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,]}) [2024-09-26 16:00:45.666136] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.666172] ERROR issue_dba_error (ob_log.cpp:1875) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.666175] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.666184] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=0] partition merge iter row count(i=0, row_count=153, ghost_row_count=0, pkey={tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337645629027, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.666205] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645665889, end_time=1727337645666205, runtime=316, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f927292fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C690-0-0, dag_ret:0, dag_status:2, start_time:1727337645663143, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.666215] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3071, dag_cnt=16, dag_cnts_[dag.get_type()]=16, &dag=0x7f927292fbd0, dag={ObIDag:{this:0x7f927292fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C690-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645663143, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.666222] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C690-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645663133, task_id:YB42C0A8002D-0006230056D8C690-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.666243] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645666242, task_id:YB42C0A8002D-0006230056D8C695-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.666246] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=3] schedule one task(task=0x7f9272931980, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.666257] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.666260] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.666262] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:45.666266] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.666269] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.666272] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=8][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.666275] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.666276] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.666278] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:45.666309] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.666305] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337645629087, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.666315] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752725472, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752725472, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.666338] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=19] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.666341] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.666346] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:215}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752725472, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.666315] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=4] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8634c80, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:516, tablet_addr:[816017-1849-0-2305843657755680768-2](offset=1925120,size=604,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:456}, data_tablet_id:{id:456}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:549484679, row_count:172}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86353a4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844548961366016-0](offset=1896448,size=1434,type=2,seq=0), type:"MAJOR", tablet_id:{id:456}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389520778000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843959477106074-0](offset=1897882,size=885,type=2,seq=0), type:"MINOR", tablet_id:{id:456}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726209424596858000, upper_trans_version:1726209424596858000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337645629102, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd2ae0, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209496670}, this:0x7f92a3bd2ae0, timestamp:1727334209496670, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:63, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277800432375000, v:0}, rec_scn:{val:1726241800152912000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697888256-2](offset=1908736,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843189604229452-0](offset=1909068,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881255412-0](offset=1909236,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447775446-0](offset=1909462,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc6790, 0x7f92a3bd2ae0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.666369] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=53] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.666361] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:215}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752725472, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:215}, table_count:1, [{i:0, table_key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645665836, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=508us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.666376] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.666382] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272931090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C694-0-0, dag_ret:0, dag_status:2, start_time:1727337645664024, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505460, output_block_cnt_arr:0x7f9266505468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337645665836, estimated_finish_time:1727337662666382}) [2024-09-26 16:00:45.666388] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272931450, type:15, status:2, dag:{ObIDag:{this:0x7f9272931090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C694-0-0, dag_ret:0, dag_status:2, start_time:1727337645664024, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.666393] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=5] task finish process(ret=0, start_time=1727337645665833, end_time=1727337645666393, runtime=560, *this={this:0x7f9272931450, type:15, status:2, dag:{ObIDag:{this:0x7f9272931090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C694-0-0, dag_ret:0, dag_status:2, start_time:1727337645664024, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.666408] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=2] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.666423] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.666426] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.666427] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:45.666431] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.666433] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864cf80, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[814042-5586-0-2305843644870074368-2](offset=1220608,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:378}, data_tablet_id:{id:378}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2166189977, row_count:350}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864d65c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844352465928192-0](offset=1212416,size=1251,type=2,seq=0), type:"MAJOR", tablet_id:{id:378}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563080818000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843914379265251-0](offset=1213667,size=843,type=2,seq=0), type:"MINI", tablet_id:{id:378}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557779816000, upper_trans_version:1726121557779816000}, {i:1, addr:[814042-5586-0-2305843914379266094-0](offset=1214510,size=843,type=2,seq=0), type:"MINI", tablet_id:{id:378}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986451466000, upper_trans_version:1726149986451466000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337645629057, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdf430, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221743022}, this:0x7f92a3bdf430, timestamp:1727334221743022, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:88, clock:641728512}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293334630000, v:0}, rec_scn:{val:1726265293334630000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513829376-2](offset=1216512,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843176718635446-0](offset=1216950,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880563282-0](offset=1217106,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447083316-0](offset=1217332,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7f2b0, 0x7f92a3bdf430, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.666472] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=39] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.666478] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.666514] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=215, range=0x7f92a9a56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:6}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685ea060, count:3, allocator:0x7f926362a7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92685ea320, local_datums:0x7f92685ea320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:45.666575] ERROR issue_dba_error (ob_log.cpp:1875) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.666577] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.666585] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=0] partition merge iter row count(i=0, row_count=9, ghost_row_count=0, pkey={tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337645629087, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.666623] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645666490, end_time=1727337645666623, runtime=133, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272931090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C694-0-0, dag_ret:0, dag_status:2, start_time:1727337645664024, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.666632] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2608, dag_cnt=15, dag_cnts_[dag.get_type()]=15, &dag=0x7f9272931090, dag={ObIDag:{this:0x7f9272931090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C694-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645664024, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.666631] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337645629102, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.666639] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C694-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645664021, task_id:YB42C0A8002D-0006230056D8C694-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.666642] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753355744, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753355744, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.666653] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645666651, task_id:YB42C0A8002D-0006230056D8C696-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.666655] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.666656] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=3] schedule one task(task=0x7f9272931eb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.666662] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.666668] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:456}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753355744, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.666683] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.666686] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.666688] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:45.666692] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.666695] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.666700] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.666685] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:456}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753355744, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:456}, table_count:1, [{i:0, table_key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8634c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645666267, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=398us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.666730] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.666740] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=10][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.666742] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:45.666745] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.666744] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=52] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92729315c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C695-0-0, dag_ret:0, dag_status:2, start_time:1727337645666246, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3490, output_block_cnt_arr:0x7f92728f3498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337645666267, estimated_finish_time:1727337662666743}) [2024-09-26 16:00:45.666748] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.666752] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241548806527000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.666750] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272931980, type:15, status:2, dag:{ObIDag:{this:0x7f92729315c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C695-0-0, dag_ret:0, dag_status:2, start_time:1727337645666246, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.666755] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=5] task finish process(ret=0, start_time=1727337645666265, end_time=1727337645666755, runtime=490, *this={this:0x7f9272931980, type:15, status:2, dag:{ObIDag:{this:0x7f92729315c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C695-0-0, dag_ret:0, dag_status:2, start_time:1727337645666246, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.666778] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=2] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.666772] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337645629057, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.666789] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.666801] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.666803] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.666809] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:378}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.666825] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:378}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:378}, table_count:1, [{i:0, table_key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645666419, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=388us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.666847] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272930630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C692-0-0, dag_ret:0, dag_status:2, start_time:1727337645663650, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5480, output_block_cnt_arr:0x7f9287db5488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337645666419, estimated_finish_time:1727337662666847}) [2024-09-26 16:00:45.666853] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f92729309f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272930630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C692-0-0, dag_ret:0, dag_status:2, start_time:1727337645663650, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.666858] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=5] task finish process(ret=0, start_time=1727337645666418, end_time=1727337645666858, runtime=440, *this={this:0x7f92729309f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272930630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C692-0-0, dag_ret:0, dag_status:2, start_time:1727337645663650, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.666866] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=2] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.666875] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337645629042, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.666893] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=14, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.666910] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.666913] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.666917] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:451}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:16, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.666931] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:451}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:451}, table_count:1, [{i:0, table_key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8650e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645666253, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=663us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.666934] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=378, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:5, allocator:0x7f926367a7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:45.666947] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272930100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C691-0-0, dag_ret:0, dag_status:2, start_time:1727337645663478, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263b1f470, output_block_cnt_arr:0x7f9263b1f478, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337645666253, estimated_finish_time:1727337662666947}) [2024-09-26 16:00:45.666952] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f92729304c0, type:15, status:2, dag:{ObIDag:{this:0x7f9272930100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C691-0-0, dag_ret:0, dag_status:2, start_time:1727337645663478, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.666956] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=4] task finish process(ret=0, start_time=1727337645666251, end_time=1727337645666956, runtime=705, *this={this:0x7f92729304c0, type:15, status:2, dag:{ObIDag:{this:0x7f9272930100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C691-0-0, dag_ret:0, dag_status:2, start_time:1727337645663478, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.666961] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.667036] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=456, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a5e060, count:7, allocator:0x7f92c1d9e7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f9264a5e3b0, local_datums:0x7f9264a5e3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:45.667048] ERROR issue_dba_error (ob_log.cpp:1875) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=18][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.667051] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.667061] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337645629057, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.667083] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645666904, end_time=1727337645667083, runtime=179, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9272930630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C692-0-0, dag_ret:0, dag_status:2, start_time:1727337645663650, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.667093] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3442, dag_cnt=14, dag_cnts_[dag.get_type()]=14, &dag=0x7f9272930630, dag={ObIDag:{this:0x7f9272930630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C692-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645663650, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.667100] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C692-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645663646, task_id:YB42C0A8002D-0006230056D8C692-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.667120] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=2] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.667127] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645667126, task_id:YB42C0A8002D-0006230056D8C697-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.667129] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=2] schedule one task(task=0x7f92729323e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.667139] ERROR issue_dba_error (ob_log.cpp:1875) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.667142] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.667153] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.667157] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.667151] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=0] partition merge iter row count(i=0, row_count=45, ghost_row_count=0, pkey={tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337645629102, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.667159] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:45.667163] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.667166] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.667172] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.667168] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645666995, end_time=1727337645667168, runtime=173, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f92729315c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C695-0-0, dag_ret:0, dag_status:2, start_time:1727337645666246, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.667177] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=1] dag finished(dag_ret=-4184, runtime=931, dag_cnt=13, dag_cnts_[dag.get_type()]=13, &dag=0x7f92729315c0, dag={ObIDag:{this:0x7f92729315c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C695-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645666246, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.667184] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C695-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645666242, task_id:YB42C0A8002D-0006230056D8C695-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.667196] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645667195, task_id:YB42C0A8002D-0006230056D8C698-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.667198] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=2] schedule one task(task=0x7f9272932910, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.667213] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337645629117, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.667221] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.667232] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.667234] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.667240] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:154}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.667268] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=24] succeed to build merge ctx(tablet_id={id:154}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:154}, table_count:1, [{i:0, table_key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8623ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645666679, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=558us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.667289] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272931af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C696-0-0, dag_ret:0, dag_status:2, start_time:1727337645666656, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db7480, output_block_cnt_arr:0x7f9287db7488, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337645666679, estimated_finish_time:1727337662667288}) [2024-09-26 16:00:45.667303] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=14] succeed to init merge ctx(task={this:0x7f9272931eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9272931af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C696-0-0, dag_ret:0, dag_status:2, start_time:1727337645666656, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.667308] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=5] task finish process(ret=0, start_time=1727337645666678, end_time=1727337645667308, runtime=630, *this={this:0x7f9272931eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9272931af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C696-0-0, dag_ret:0, dag_status:2, start_time:1727337645666656, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.667321] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=2] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.667336] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337645629133, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.667346] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.667356] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.667359] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.667365] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101015}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.667373] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=154, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5e060, count:5, allocator:0x7f92ba7fc7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264d5e360, local_datums:0x7f9264d5e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:45.667381] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101015}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101015}, table_count:1, [{i:0, table_key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645667149, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=214us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.667409] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=22] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272932020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C697-0-0, dag_ret:0, dag_status:2, start_time:1727337645667129, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d490, output_block_cnt_arr:0x7f9264b1d498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337645667149, estimated_finish_time:1727337662667409}) [2024-09-26 16:00:45.667415] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f92729323e0, type:15, status:2, dag:{ObIDag:{this:0x7f9272932020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C697-0-0, dag_ret:0, dag_status:2, start_time:1727337645667129, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.667419] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=4] task finish process(ret=0, start_time=1727337645667146, end_time=1727337645667419, runtime=273, *this={this:0x7f92729323e0, type:15, status:2, dag:{ObIDag:{this:0x7f9272932020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C697-0-0, dag_ret:0, dag_status:2, start_time:1727337645667129, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.667438] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=3] schedule one task(task=0x7f927282a3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.667478] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.667473] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101015, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:6, allocator:0x7f92865547d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:45.667482] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.667517] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337645629117, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.667535] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645667347, end_time=1727337645667535, runtime=188, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9272931af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C696-0-0, dag_ret:0, dag_status:2, start_time:1727337645666656, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.667543] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=887, dag_cnt=12, dag_cnts_[dag.get_type()]=12, &dag=0x7f9272931af0, dag={ObIDag:{this:0x7f9272931af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C696-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645666656, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.667550] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C696-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645666651, task_id:YB42C0A8002D-0006230056D8C696-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.667565] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.667578] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=13][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.667587] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645667586, task_id:YB42C0A8002D-0006230056D8C699-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.667590] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=2] schedule one task(task=0x7f9272932e40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.667587] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337645629133, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.667605] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645667447, end_time=1727337645667605, runtime=158, *this={this:0x7f927282a3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9272932020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C697-0-0, dag_ret:0, dag_status:2, start_time:1727337645667129, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.667613] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=484, dag_cnt=11, dag_cnts_[dag.get_type()]=11, &dag=0x7f9272932020, dag={ObIDag:{this:0x7f9272932020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C697-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645667129, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.667620] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C697-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645667126, task_id:YB42C0A8002D-0006230056D8C697-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.667632] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645667631, task_id:YB42C0A8002D-0006230056D8C69A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.667634] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=2] schedule one task(task=0x7f9272933370, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.667665] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.667668] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.667670] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:45.667674] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.667677] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.667668] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=451, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:14, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:16}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92728f8060, count:4, allocator:0x7f92691a27d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:16, datum_buffer:{capacity:32, datums:0x7f92728f8350, local_datums:0x7f92728f8350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,]}) [2024-09-26 16:00:45.667682] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.667719] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.667721] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.667729] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337645629042, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.667744] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645667641, end_time=1727337645667744, runtime=103, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272930100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C691-0-0, dag_ret:0, dag_status:2, start_time:1727337645663478, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.667752] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4273, dag_cnt=10, dag_cnts_[dag.get_type()]=10, &dag=0x7f9272930100, dag={ObIDag:{this:0x7f9272930100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C691-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645663478, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.667758] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C691-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337645663472, task_id:YB42C0A8002D-0006230056D8C691-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.667770] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645667768, task_id:YB42C0A8002D-0006230056D8C69B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.667772] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=2] schedule one task(task=0x7f92729338a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.667786] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.667789] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.667791] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:45.667794] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.667798] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b864fe60, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:510, tablet_addr:[816017-1849-0-2305843651313397760-2](offset=2093056,size=598,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101063}, data_tablet_id:{id:297}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726063409221604000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1864622635, row_count:24}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86505a8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844568288894976-0](offset=2072576,size=1452,type=2,seq=0), type:"MAJOR", tablet_id:{id:101063}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063202518238000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843928338769324-0](offset=2074028,size=856,type=2,seq=0), type:"MINI", tablet_id:{id:101063}, scn_range:{start_scn:{val:1726063409221604000, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557514739000, upper_trans_version:1726121557514739000}, {i:1, addr:[816017-1849-0-2305843928338770180-0](offset=2074884,size=856,type=2,seq=0), type:"MINI", tablet_id:{id:101063}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149603136610000, upper_trans_version:1726149603136610000}, {i:2, addr:[816017-1849-0-2305843927265029212-0](offset=2075740,size=855,type=2,seq=0), type:"MINI", tablet_id:{id:101063}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726207962418314000, upper_trans_version:1726207962418314000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337645629163, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd8ea0, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220519573}, this:0x7f92a3bd8ea0, timestamp:1727334220519573, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:75, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726264804632655000, v:0}, rec_scn:{val:1726264804632655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843593331331072-2](offset=2084864,size=544,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046856736-0](offset=2085408,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881431758-0](offset=2085582,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bca610, 0x7f92a3bd8ea0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.667836] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=38] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.667842] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.667862] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.667866] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.667867] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:45.667870] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.667885] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337645629180, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.667872] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706139416032, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f926e072bc0, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[812168-2208-0-2305843644869693440-2](offset=839680,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:328}, data_tablet_id:{id:328}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706139416032, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:867269498, row_count:786}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f926e07329c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[812168-2208-0-2305844589762490368-0](offset=831488,size=1472,type=2,seq=0), type:"MAJOR", tablet_id:{id:328}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562828712000}}, minor_tables:{{i:0, addr:[812168-2208-0-2305843935853721024-0](offset=832960,size=863,type=2,seq=0), type:"MINI", tablet_id:{id:328}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986144852000, upper_trans_version:1726149986144852000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337645629196, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bda610, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220621826}, this:0x7f92a3bda610, timestamp:1727334220621826, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:78, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265257450254001, v:0}, rec_scn:{val:1726264907253071001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[812168-2208-0-2305843366770556928-2](offset=835584,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[812168-2208-0-2305843176718254413-0](offset=835917,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843251880182249-0](offset=836073,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843216446702283-0](offset=836299,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc8ea0, 0x7f92a3bda610, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.667897] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f92728f7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=15, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.667903] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=31] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.667907] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.667908] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.667911] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.667930] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=16] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:452}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:17, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.667920] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752511928, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.667947] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=23] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.667950] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.667946] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:452}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f92728f7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:452}, table_count:1, [{i:0, table_key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8652d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645667661, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=267us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.667955] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:121}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752511928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.667980] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=28] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272932fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69A-0-0, dag_ret:0, dag_status:2, start_time:1727337645667634, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f7480, output_block_cnt_arr:0x7f92728f7488, concurrent_cnt:1, estimate_row_cnt:10, estimate_occupy_size:0, latest_update_ts:1727337645667661, estimated_finish_time:1727337662667979}) [2024-09-26 16:00:45.667986] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272933370, type:15, status:2, dag:{ObIDag:{this:0x7f9272932fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69A-0-0, dag_ret:0, dag_status:2, start_time:1727337645667634, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.667991] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=4] task finish process(ret=0, start_time=1727337645667659, end_time=1727337645667990, runtime=331, *this={this:0x7f9272933370, type:15, status:2, dag:{ObIDag:{this:0x7f9272932fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69A-0-0, dag_ret:0, dag_status:2, start_time:1727337645667634, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.667981] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=20] succeed to build merge ctx(tablet_id={id:121}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645666726, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.23ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.668005] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C693-0-0, dag_ret:0, dag_status:2, start_time:1727337645663887, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f9263a45480, output_block_cnt_arr:0x7f9263a45488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337645666726, estimated_finish_time:1727337662668004}) [2024-09-26 16:00:45.668022] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=17] succeed to init merge ctx(task={this:0x7f9272930f20, type:15, status:2, dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C693-0-0, dag_ret:0, dag_status:2, start_time:1727337645663887, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:45.668024] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=2] schedule one task(task=0x7f9272934ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.668027] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=5] task finish process(ret=0, start_time=1727337645666725, end_time=1727337645668027, runtime=1302, *this={this:0x7f9272930f20, type:15, status:2, dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C693-0-0, dag_ret:0, dag_status:2, start_time:1727337645663887, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:45.668046] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=5] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.668074] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=452, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:15, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:17}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ec060, count:5, allocator:0x7f9275d767d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:17, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,]}) [2024-09-26 16:00:45.668112] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.668144] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.668147] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.668156] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=0] partition merge iter row count(i=0, row_count=10, ghost_row_count=0, pkey={tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337645629180, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.668162] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=13] multi version scan iterator init succ(param.table_id_=121, range=0x7f92a9ed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a60060, count:5, allocator:0x7f92c1d9e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9264a60360, local_datums:0x7f9264a60360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:45.668173] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337645668037, end_time=1727337645668173, runtime=136, *this={this:0x7f9272934ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9272932fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69A-0-0, dag_ret:0, dag_status:2, start_time:1727337645667634, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.668178] INFO [STORAGE] dump_info (ob_sstable_struct.cpp:262) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=12] dump merge info(msg="macro block builder close", output_row_per_s=0, new_macro_KB_per_s=0, *this={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:38, merge_start_time:1727337645668139, merge_finish_time:1727337645668177, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:0, max:0, sum:0, count:0}}, {type:"cost_time", info:{min:0, max:0, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:0, max:0, sum:0, count:0}}, {type:"incremental_row_count", info:{min:0, max:0, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:0, snapshot_version:0, start_scn:0, end_scn:0}, macro_id_list:"", comment:""}) [2024-09-26 16:00:45.668199] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=548, dag_cnt=9, dag_cnts_[dag.get_type()]=9, &dag=0x7f9272932fb0, dag={ObIDag:{this:0x7f9272932fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645667634, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.668208] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69A-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337645667631, task_id:YB42C0A8002D-0006230056D8C69A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.668205] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=26] partition merge iter row count(i=0, row_count=0, ghost_row_count=0, pkey={tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, table={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337645629072, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.668217] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1544) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=9] merge macro blocks ok(idx_=0, task={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C693-0-0, dag_ret:0, dag_status:2, start_time:1727337645663887, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:45.668222] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645668220, task_id:YB42C0A8002D-0006230056D8C69C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.668226] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=4] schedule one task(task=0x7f9272933dd0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.668227] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=6] task finish process(ret=0, start_time=1727337645668136, end_time=1727337645668227, runtime=91, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C693-0-0, dag_ret:0, dag_status:2, start_time:1727337645663887, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:45.668244] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=2] schedule one task(task=0x7f926585e250, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.668279] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.668282] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.668284] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:45.668287] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.668290] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706539141067, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f926e107fe0, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:2, version:2, length:516, tablet_addr:[809012-1445-0-2305843657754198016-2](offset=442368,size=604,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:330}, data_tablet_id:{id:330}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706539141067, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3803410633, row_count:15330}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f926e108704, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844604794474496-0](offset=430080,size=1486,type=2,seq=0), type:"MAJOR", tablet_id:{id:330}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562805980000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843948738221518-0](offset=431566,size=875,type=2,seq=0), type:"MINI", tablet_id:{id:330}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986144852000, upper_trans_version:1726149986144852000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337645629226, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdade0, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220705914}, this:0x7f92a3bdade0, timestamp:1727334220705914, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:79, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265269081598001, v:0}, rec_scn:{val:1726264907253071001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696413696-2](offset=434176,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843189602754892-0](offset=434508,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879780852-0](offset=434676,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446300886-0](offset=434902,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc9670, 0x7f92a3bdade0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:45.668325] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=35] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.668330] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.668343] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337645629196, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.668351] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=7] update column cnt(column_cnt_=26, store_column_cnt_=26, column_cnt_=26, column_array_.count()=0) [2024-09-26 16:00:45.668354] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9263a43480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=26, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.668364] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.668366] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.668370] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:328}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:28, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.668386] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:328}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9263a43480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:328}, table_count:1, [{i:0, table_key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e072bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645667858, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=511us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.668406] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92729334e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69B-0-0, dag_ret:0, dag_status:2, start_time:1727337645667772, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a435d0, output_block_cnt_arr:0x7f9263a435d8, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337645667858, estimated_finish_time:1727337662668406}) [2024-09-26 16:00:45.668422] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=16] succeed to init merge ctx(task={this:0x7f92729338a0, type:15, status:2, dag:{ObIDag:{this:0x7f92729334e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69B-0-0, dag_ret:0, dag_status:2, start_time:1727337645667772, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.668427] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=5] task finish process(ret=0, start_time=1727337645667858, end_time=1727337645668427, runtime=569, *this={this:0x7f92729338a0, type:15, status:2, dag:{ObIDag:{this:0x7f92729334e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69B-0-0, dag_ret:0, dag_status:2, start_time:1727337645667772, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.668436] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=3] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.668454] INFO [STORAGE.COMPACTION] get_merged_sstable (ob_tablet_merge_task.cpp:1285) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=2] create new merged sstable(ctx.param_.tablet_id_={id:121}, snapshot_version=1726241132764173999, ctx.param_.merge_type_=3, ctx.create_snapshot_version_=0, table_mode_flag=0) [2024-09-26 16:00:45.668461] INFO [STORAGE] close (ob_index_block_builder.cpp:815) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=3] succeed to close sstable index builder(res={root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_block_ids_.count():0, other_block_ids_.count():0, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, data_column_cnt:10, data_column_checksums:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], row_count:0, max_merged_trans_version:0, contain_uncommitted_row:false, occupy_size:0, original_size:0, data_checksum:0, use_old_macro_block_count:0, compressor_type:1, root_row_store_type:0, nested_offset:0, nested_size:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}) [2024-09-26 16:00:45.668473] INFO [STORAGE] init (ob_sstable.cpp:118) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=7] succeeded to init sstable(ret=0, this={ObITable:{this:0x7f9263a44128, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9263a44128, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:null, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:45.668493] INFO [STORAGE.COMPACTION] create_sstable (ob_tablet_merge_ctx.cpp:367) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=19] succeed to merge sstable(param={table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_logic_seq:0, schema_version:1710828752511928, create_snapshot_version:0, progressive_merge_round:0, progressive_merge_step:0, is_ready_for_read:true, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, index_type:0, root_block_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), root_block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, root_row_store_type:0, latest_row_store_type:1, data_index_tree_height:0, data_block_macro_meta_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), data_block_macro_meta:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, use_old_macro_block_count:0, row_count:0, rowkey_column_cnt:5, column_cnt:10, column_checksums:[], data_checksum:0, occupy_size:0, original_size:0, max_merged_trans_version:1726241132764173999, ddl_scn:{val:0, v:0}, contain_uncommitted_row:false, is_meta_root:false, compressor_type:1, encrypt_id:0, master_key_id:0, recycle_version:0, nested_offset:0, nested_size:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}, table_key={tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_merge_info={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:1451, merge_start_time:1727337645666726, merge_finish_time:1727337645668177, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"cost_time", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"incremental_row_count", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:1, snapshot_version:0, start_scn:1726211867305630000, end_scn:1726241548806527000}, macro_id_list:"", comment:""}) [2024-09-26 16:00:45.668592] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=328, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:26, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:28}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b2060, count:5, allocator:0x7f92865b67d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:28, datum_buffer:{capacity:32, datums:0x7f92684b2360, local_datums:0x7f92684b2360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,]}) [2024-09-26 16:00:45.668655] ERROR issue_dba_error (ob_log.cpp:1875) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.668658] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.668665] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337645629196, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.668703] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645668567, end_time=1727337645668702, runtime=135, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f92729334e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69B-0-0, dag_ret:0, dag_status:2, start_time:1727337645667772, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.668713] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=941, dag_cnt=8, dag_cnts_[dag.get_type()]=8, &dag=0x7f92729334e0, dag={ObIDag:{this:0x7f92729334e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645667772, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.668720] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645667768, task_id:YB42C0A8002D-0006230056D8C69B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.668733] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337645668732, task_id:YB42C0A8002D-0006230056D8C69D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.668737] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=4] schedule one task(task=0x7f9272934300, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.668754] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.668757] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.668752] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337645629226, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.668759] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:45.668760] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=8] update column cnt(column_cnt_=12, store_column_cnt_=12, column_cnt_=12, column_array_.count()=0) [2024-09-26 16:00:45.668763] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.668767] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.668763] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92684ab490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.668772] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.668782] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.668785] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.668789] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:330}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.668807] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:330}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92684ab490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:330}, table_count:1, [{i:0, table_key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e107fe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645668275, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=512us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.668833] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272933a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69C-0-0, dag_ret:0, dag_status:2, start_time:1727337645668226, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab5f0, output_block_cnt_arr:0x7f92684ab5f8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337645668275, estimated_finish_time:1727337662668833}) [2024-09-26 16:00:45.668839] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272933dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9272933a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69C-0-0, dag_ret:0, dag_status:2, start_time:1727337645668226, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.668843] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=4] task finish process(ret=0, start_time=1727337645668273, end_time=1727337645668843, runtime=570, *this={this:0x7f9272933dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9272933a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69C-0-0, dag_ret:0, dag_status:2, start_time:1727337645668226, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.668850] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=2] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.668897] INFO [STORAGE] build_new_table_store (ob_tablet_table_store.cpp:1206) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=19] succeed to build new table store(major_tables_={cnt:1, is_inited:true}, minor_tables_={cnt:3, is_inited:true}, memtables_={count:1}, PRINT_TS(*this)={ObTabletTableStore_Pretty:this:0x7f92a9d51ca8, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, memtables:{count:1}, is_ready_for_read:true, table_array:{ [2882] [ YB42C0A8002D-0006230056D8C693-0-0 ] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [2882] [ YB42C0A8002D-0006230056D8C693-0-0 ] MAJOR MAJOR 1725120003420849000 1725120003420849000 {val:0, v:0} {val:1726077600415496000, v:0} 0 false [2882] [ YB42C0A8002D-0006230056D8C693-0-0 ] MINOR MINOR 1726178585269925002 1726178585269925002 {val:1726077259159067000, v:0} {val:1726178585269925002, v:0} 0 false [2882] [ YB42C0A8002D-0006230056D8C693-0-0 ] MINI 1726211867106157000 1726211867106157000 {val:1726178585269925002, v:0} {val:1726211867305630000, v:0} 0 false [2882] [ YB42C0A8002D-0006230056D8C693-0-0 ] MINI 1726241132764173999 1726241132764173999 {val:1726211867305630000, v:0} {val:1726241548806527000, v:0} 0 false [2882] [ YB42C0A8002D-0006230056D8C693-0-0 ] MEM ACTIVE 9223372036854775807 18446744073709547609 {val:1726241548806527000, v:0} {val:4611686018427387903, v:0} 3 unused }}) [2024-09-26 16:00:45.668935] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337645629242, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.668946] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=10] update column cnt(column_cnt_=29, store_column_cnt_=29, column_cnt_=29, column_array_.count()=0) [2024-09-26 16:00:45.668948] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92684ad490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.668964] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=15] multi version scan iterator init succ(param.table_id_=330, range=0x7f92aa352200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d32060, count:7, allocator:0x7f9269f787d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9287d323b0, local_datums:0x7f9287d323b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:45.668985] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.668990] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.668996] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:329}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.669015] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:329}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92684ad490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:329}, table_count:1, [{i:0, table_key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e089f20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645668749, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=244us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.669040] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272933f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69D-0-0, dag_ret:0, dag_status:2, start_time:1727337645668737, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad5f0, output_block_cnt_arr:0x7f92684ad5f8, concurrent_cnt:1, estimate_row_cnt:62, estimate_occupy_size:0, latest_update_ts:1727337645668749, estimated_finish_time:1727337662669040}) [2024-09-26 16:00:45.669046] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272934300, type:15, status:2, dag:{ObIDag:{this:0x7f9272933f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69D-0-0, dag_ret:0, dag_status:2, start_time:1727337645668737, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.669051] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=5] task finish process(ret=0, start_time=1727337645668747, end_time=1727337645669051, runtime=304, *this={this:0x7f9272934300, type:15, status:2, dag:{ObIDag:{this:0x7f9272933f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69D-0-0, dag_ret:0, dag_status:2, start_time:1727337645668737, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.669066] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=3] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.669135] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=20][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.669169] ERROR issue_dba_error (ob_log.cpp:1875) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.669171] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.669181] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=0] partition merge iter row count(i=0, row_count=141, ghost_row_count=0, pkey={tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337645629226, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.669198] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645668937, end_time=1727337645669198, runtime=261, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9272933a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69C-0-0, dag_ret:0, dag_status:2, start_time:1727337645668226, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.669207] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=981, dag_cnt=7, dag_cnts_[dag.get_type()]=7, &dag=0x7f9272933a10, dag={ObIDag:{this:0x7f9272933a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645668226, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.669226] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69C-0-0] [lt=17] succeed to del sys task(removed_task={start_time:1727337645668220, task_id:YB42C0A8002D-0006230056D8C69C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.669240] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645669238, task_id:YB42C0A8002D-0006230056D8C69E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.669243] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=3] schedule one task(task=0x7f9272934830, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.669268] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.669272] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.669274] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:45.669279] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.669282] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.669273] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=329, range=0x7f92a9952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fa060, count:6, allocator:0x7f92b32407d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f92665fa390, local_datums:0x7f92665fa390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:45.669288] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.669379] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.669424] ERROR issue_dba_error (ob_log.cpp:1875) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.669427] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.669436] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=0] partition merge iter row count(i=0, row_count=47, ghost_row_count=0, pkey={tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337645629242, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.669452] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645669248, end_time=1727337645669452, runtime=204, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272933f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69D-0-0, dag_ret:0, dag_status:2, start_time:1727337645668737, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.669473] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=735, dag_cnt=6, dag_cnts_[dag.get_type()]=6, &dag=0x7f9272933f40, dag={ObIDag:{this:0x7f9272933f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645668737, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.669488] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69D-0-0] [lt=13] succeed to del sys task(removed_task={start_time:1727337645668732, task_id:YB42C0A8002D-0006230056D8C69D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.669523] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337645669521, task_id:YB42C0A8002D-0006230056D8C69F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.669516] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337645629257, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.669526] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=2] schedule one task(task=0x7f9272934d60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.669527] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92665f9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=25, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.669539] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.669542] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.669552] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.669556] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.669558] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:45.669547] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92665f9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:332}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:27, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.669562] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.669565] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.669572] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.669565] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:332}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92665f9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:332}, table_count:1, [{i:0, table_key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8648b00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645669263, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=282us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.669587] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337645629163, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.669597] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=26] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272934470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69E-0-0, dag_ret:0, dag_status:2, start_time:1727337645669243, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665f9490, output_block_cnt_arr:0x7f92665f9498, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337645669263, estimated_finish_time:1727337662669596}) [2024-09-26 16:00:45.669596] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.669603] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272934830, type:15, status:2, dag:{ObIDag:{this:0x7f9272934470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69E-0-0, dag_ret:0, dag_status:2, start_time:1727337645669243, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.669607] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.669609] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.669608] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=5] task finish process(ret=0, start_time=1727337645669261, end_time=1727337645669608, runtime=347, *this={this:0x7f9272934830, type:15, status:2, dag:{ObIDag:{this:0x7f9272934470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69E-0-0, dag_ret:0, dag_status:2, start_time:1727337645669243, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.669617] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=2] schedule one task(task=0x7f926585e3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.669614] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101063}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.669632] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101063}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101063}, table_count:1, [{i:0, table_key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645667782, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.83ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.669653] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272932a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C699-0-0, dag_ret:0, dag_status:2, start_time:1727337645667590, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665054a0, output_block_cnt_arr:0x7f92665054a8, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337645667782, estimated_finish_time:1727337662669653}) [2024-09-26 16:00:45.669659] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272932e40, type:15, status:2, dag:{ObIDag:{this:0x7f9272932a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C699-0-0, dag_ret:0, dag_status:2, start_time:1727337645667590, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.669664] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=5] task finish process(ret=0, start_time=1727337645667781, end_time=1727337645669664, runtime=1883, *this={this:0x7f9272932e40, type:15, status:2, dag:{ObIDag:{this:0x7f9272932a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C699-0-0, dag_ret:0, dag_status:2, start_time:1727337645667590, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.669655] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=332, range=0x7f92a9cd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:25, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:27}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685e8060, count:6, allocator:0x7f9275d787d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:27, datum_buffer:{capacity:32, datums:0x7f92685e8390, local_datums:0x7f92685e8390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,]}) [2024-09-26 16:00:45.669687] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337645628996, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.669695] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.669710] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.669712] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.669726] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=11] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:333}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.669743] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:333}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:333}, table_count:1, [{i:0, table_key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8651da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645663178, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=6.55ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.669775] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927292f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68E-0-0, dag_ret:0, dag_status:2, start_time:1727337645662842, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1490, output_block_cnt_arr:0x7f92728f1498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337645663178, estimated_finish_time:1727337662669775}) [2024-09-26 16:00:45.669782] ERROR issue_dba_error (ob_log.cpp:1875) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.669781] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927292f530, type:15, status:2, dag:{ObIDag:{this:0x7f927292f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68E-0-0, dag_ret:0, dag_status:2, start_time:1727337645662842, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.669786] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=5] task finish process(ret=0, start_time=1727337645663176, end_time=1727337645669786, runtime=6610, *this={this:0x7f927292f530, type:15, status:2, dag:{ObIDag:{this:0x7f927292f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68E-0-0, dag_ret:0, dag_status:2, start_time:1727337645662842, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.669785] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.669794] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=1] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337645629257, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.669804] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337645629272, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.669812] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.669815] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645669626, end_time=1727337645669815, runtime=189, *this={this:0x7f926585e3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9272934470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69E-0-0, dag_ret:0, dag_status:2, start_time:1727337645669243, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.669827] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.669824] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=581, dag_cnt=5, dag_cnts_[dag.get_type()]=5, &dag=0x7f9272934470, dag={ObIDag:{this:0x7f9272934470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645669243, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.669832] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69E-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337645669238, task_id:YB42C0A8002D-0006230056D8C69E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.669833] INFO [MDS] init_for_evict_medium_info (ob_tablet_mds_data.cpp:414) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=14] succeeded to init mds data(ret=0, finish_medium_scn=1726077600415496000, src_medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, src_extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}) [2024-09-26 16:00:45.669853] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.669858] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=2] schedule one task(task=0x7f9272934ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.669894] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101063, range=0x7f92ab152200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:8, allocator:0x7f92ba7fc7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264a623e0, local_datums:0x7f9264a623e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:45.669930] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=26][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:45.669857] INFO [STORAGE] init_for_merge (ob_tablet.cpp:363) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=6] succeeded to init tablet for mini/minor/major merge(ret=0, param={sstable:0x7f9263a44128, snapshot_version:1726241132764173999, clog_checkpoint_scn:{val:1726241548806527000, v:0}, multi_version_start:1726178585269925002, need_report:false, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, rebuild_seq:0, update_with_major_flag:false, need_check_sstable:false, ddl_info:{keep_old_ddl_sstable:true, ddl_start_scn:{val:0, v:0}, ddl_commit_scn:{val:0, v:0}, ddl_checkpoint_scn:{val:0, v:0}, ddl_snapshot_version:0, ddl_execution_id:-1, data_format_version:0}, allow_duplicate_sstable:false, merge_type:"MINI_MERGE", need_check_transfer_seq:true, transfer_seq:0}, old_tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337645628266000, hold_ref_cnt:true, ref_cnt:3, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337645629072, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}, this={this:0x7f92728fa060, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x7f92a9d52450, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726241548806527000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726241132764173999, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92728fa670, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}, {i:2, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726241132764173999, upper_trans_version:1726241132764173999}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x7f92728fa520, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f92728fb880, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f92728fb950, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{redefined:false, snapshot_version:0, schema_version:0, data_tablet_id:{id:0}, hidden_tablet_id:{id:0}, lob_meta_tablet_id:{id:50121}, lob_piece_tablet_id:{id:60121}, is_old_mds:true}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:45.669972] ERROR issue_dba_error (ob_log.cpp:1875) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.669975] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.669981] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:45.669984] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:45.669978] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=107] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f92728f6060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}}, ref_cnt:0, upper_trans_version:1725120003420849000, timestamp:0}, this:0x7f92728f6060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1725120003420849000, max_merged_trans_version:1725120003420849000, data_macro_block_count:1, nested_size:4096, nested_offset:589824, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:84, row_count_:14, occupy_size_:1604, original_size_:1914, data_checksum_:3587602083, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:1, data_micro_block_count_:1, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:2, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1725120003420849000, max_merged_trans_version_:1725120003420849000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f92728f6310, column_checksum_count:10, data_root_info:{addr:[0-0-0-3458764933653594112-0](offset=0,size=391,type=3,seq=0), block_data:{buf:0x7f92728f6360, size:391, extra_buf:0x7f92728f64e7, extra_size:424, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-3458765009889263616-0](offset=0,size=462,type=3,seq=0), block_data:{buf:0x7f92728f668f, size:462, extra_buf:null, extra_size:0, type:1}}, data_block_count:1, other_block_count:0, linked_block_count:0, data_block_ids:0x7f92728f685d, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:true, nested_offset:589824, nested_size:4096}}, valid_for_reading:true}) [2024-09-26 16:00:45.669985] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:8, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32070, hex: F1BA040000000000, int: 310001},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d320a8, hex: 0300000000000000, int: 3},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d320e0, hex: 0000000000000000, int: 0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32118, hex: 0500000000000000, int: 5},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32150, hex: 20BB040000000000, int: 310048},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32188, hex: 0000000000000000, int: 0},idx=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d321c0, hex: 8097ACF17E2D0BE8, int: -1726236008922704000},idx=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d321f8, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:7654, column_count:8, rowkey_column_count:8, has_column_checksum:0, row_count:1, row_store_type:0, opt:5, var_column_count:0, row_offset:116, original_length:60, max_merged_trans_version:1726236008922704000, data_length:60, data_zlength:60, data_checksum:908626827, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92b6804090, buf_size:60, data_size:60, row_count:1, column_count:8, max_merged_trans_version:1726236008922704000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2087180190, row_count_delta:1, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:60}) [2024-09-26 16:00:45.670013] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=27][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:45.670005] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=25] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f92728f6060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}}, ref_cnt:0, upper_trans_version:1726178585269925002, timestamp:0}, this:0x7f92728f6060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726178585269925002, max_merged_trans_version:1726178585269925002, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726178585269925002, v:0}, meta:{basic_meta:{version:1, length:94, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726178585269925002, max_merged_trans_version_:1726178585269925002, recycle_version:1726077600415496000, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726178585269925002, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:1, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f92728f6310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:45.670015] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=1][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:00:45.670016] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=1][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:00:45.670018] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=2][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:00:45.670019] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=1][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:00:45.670017] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=11] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f92728f6060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}}, ref_cnt:0, upper_trans_version:1726211867106157000, timestamp:0}, this:0x7f92728f6060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726211867106157000, max_merged_trans_version:1726211867106157000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726211867305630000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726211867106157000, max_merged_trans_version_:1726211867106157000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726211867305630000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f92728f6310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:45.670021] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=2] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337645629163, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.670027] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=9] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f92728f6060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f92728f6060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f92728f6310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:45.670034] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=10][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:45.670041] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:45.670044] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:45.670046] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337645669868, end_time=1727337645670046, runtime=178, *this={this:0x7f9272934ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9272932a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C699-0-0, dag_ret:0, dag_status:2, start_time:1727337645667590, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.670051] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=5][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9272934ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9272932a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C699-0-0, dag_ret:0, dag_status:2, start_time:1727337645667590, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:45.670058] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=2468, dag_cnt=4, dag_cnts_[dag.get_type()]=4, &dag=0x7f9272932a80, dag={ObIDag:{this:0x7f9272932a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C699-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645667590, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.670065] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C699-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645667586, task_id:YB42C0A8002D-0006230056D8C699-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.670068] ERROR issue_dba_error (ob_log.cpp:1875) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.670070] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf9de5b1 0xf9dcfd6 0xe51c6fb 0xe519c03 0xe518f89 0xe516f8c 0xf01158d 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.670101] INFO [STORAGE] reset (ob_tablet.cpp:159) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=1] reset tablet(this=0x7f92728fa060, ls_id={id:1}, tablet_id={id:121}, lbt()="0x121f1f9c 0xe52cc09 0xe52c6c5 0xe52bd13 0x4fd348f 0xf01110c 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:45.670109] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1250) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=0][errcode=-4184] sstable merge finish(ret=-4184, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:true, merge_dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C693-0-0, dag_ret:0, dag_status:2, start_time:1727337645663887, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f9263a45480, output_block_cnt_arr:0x7f9263a45488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337645666726, estimated_finish_time:1727337662668004}, compaction_filter:NULL, time_guard:total=1.79ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, task={this:0x7f926585e250, type:16, status:2, dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C693-0-0, dag_ret:0, dag_status:2, start_time:1727337645663887, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:45.670133] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645668452, end_time=1727337645670132, runtime=1680, *this={this:0x7f926585e250, type:16, status:2, dag:{ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C693-0-0, dag_ret:0, dag_status:2, start_time:1727337645663887, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:45.670139] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=6252, dag_cnt=3, dag_cnts_[dag.get_type()]=3, &dag=0x7f9272930b60, dag={ObIDag:{this:0x7f9272930b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C693-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645663887, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}) [2024-09-26 16:00:45.670146] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C693-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645663883, task_id:YB42C0A8002D-0006230056D8C693-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.670209] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=333, range=0x7f92aa6d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:7, allocator:0x7f9275d767d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f92684b43b0, local_datums:0x7f92684b43b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:45.670358] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:45.670390] ERROR issue_dba_error (ob_log.cpp:1875) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.670392] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.670401] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=0] partition merge iter row count(i=0, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337645628996, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.670419] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645670182, end_time=1727337645670419, runtime=237, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f927292f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68E-0-0, dag_ret:0, dag_status:2, start_time:1727337645662842, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.670427] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=7585, dag_cnt=2, dag_cnts_[dag.get_type()]=2, &dag=0x7f927292f170, dag={ObIDag:{this:0x7f927292f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C68E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645662842, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.670433] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C68E-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645662838, task_id:YB42C0A8002D-0006230056D8C68E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.670485] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.670493] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101067}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.670513] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:101067}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101067}, table_count:1, [{i:0, table_key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b86a2d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645669547, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=943us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.670542] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92729349a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69F-0-0, dag_ret:0, dag_status:2, start_time:1727337645669526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a434a0, output_block_cnt_arr:0x7f9263a434a8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337645669547, estimated_finish_time:1727337662670542}) [2024-09-26 16:00:45.670548] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272934d60, type:15, status:2, dag:{ObIDag:{this:0x7f92729349a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69F-0-0, dag_ret:0, dag_status:2, start_time:1727337645669526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.670553] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=5] task finish process(ret=0, start_time=1727337645669546, end_time=1727337645670553, runtime=1007, *this={this:0x7f9272934d60, type:15, status:2, dag:{ObIDag:{this:0x7f92729349a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69F-0-0, dag_ret:0, dag_status:2, start_time:1727337645669526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.670562] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=2] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.670595] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.670598] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:45.670601] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:45.670605] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:45.670608] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:45.670614] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.670653] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101067, range=0x7f92aaf52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:8, allocator:0x7f92b321a7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664f03e0, local_datums:0x7f92664f03e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:45.670870] ERROR issue_dba_error (ob_log.cpp:1875) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.670873] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.670880] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=0] partition merge iter row count(i=0, row_count=227, ghost_row_count=0, pkey={tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337645629272, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.670899] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645670629, end_time=1727337645670899, runtime=270, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f92729349a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69F-0-0, dag_ret:0, dag_status:2, start_time:1727337645669526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.670907] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1381, dag_cnt=1, dag_cnts_[dag.get_type()]=1, &dag=0x7f92729349a0, dag={ObIDag:{this:0x7f92729349a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C69F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645669526, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.670913] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C69F-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337645669521, task_id:YB42C0A8002D-0006230056D8C69F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.670962] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337645629148, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:45.670971] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:45.670983] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:45.670986] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:45.670990] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101016}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:45.671007] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:101016}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101016}, table_count:1, [{i:0, table_key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337645670590, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=399us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:45.671028] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272932550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C698-0-0, dag_ret:0, dag_status:2, start_time:1727337645667198, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5490, output_block_cnt_arr:0x7f9287db5498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337645670590, estimated_finish_time:1727337662671028}) [2024-09-26 16:00:45.671034] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272932910, type:15, status:2, dag:{ObIDag:{this:0x7f9272932550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C698-0-0, dag_ret:0, dag_status:2, start_time:1727337645667198, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.671039] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=5] task finish process(ret=0, start_time=1727337645670589, end_time=1727337645671039, runtime=450, *this={this:0x7f9272932910, type:15, status:2, dag:{ObIDag:{this:0x7f9272932550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C698-0-0, dag_ret:0, dag_status:2, start_time:1727337645667198, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.671046] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=2] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:45.671100] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=101016, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d36060, count:6, allocator:0x7f92636287d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9287d36390, local_datums:0x7f9287d36390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:45.671189] ERROR issue_dba_error (ob_log.cpp:1875) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:45.671192] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:45.671200] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=1] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337645629148, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:45.671217] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337645671074, end_time=1727337645671217, runtime=143, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9272932550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C698-0-0, dag_ret:0, dag_status:2, start_time:1727337645667198, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:45.671226] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4028, dag_cnt=0, dag_cnts_[dag.get_type()]=0, &dag=0x7f9272932550, dag={ObIDag:{this:0x7f9272932550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C698-0-0, dag_ret:-4184, dag_status:5, start_time:1727337645667198, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:45.671244] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C698-0-0] [lt=16] succeed to del sys task(removed_task={start_time:1727337645667195, task_id:YB42C0A8002D-0006230056D8C698-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:45.685336] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.685345] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.697351] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:45.715516] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.715524] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.720287] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:45.720302] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337645720284) [2024-09-26 16:00:45.720307] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337645620233, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:45.720324] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223374955770, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.720328] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:45.720335] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:45.720340] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:45.720342] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:45.720348] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.720350] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.720352] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337645720310) [2024-09-26 16:00:45.735517] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=9] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:45.745902] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.745911] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.754012] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337645320844]}) [2024-09-26 16:00:45.765398] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:45.776220] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:45.776230] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.806328] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.806335] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.806662] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337645806662, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006880, add_bytes: 0 [2024-09-26 16:00:45.820314] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223375055924, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.820324] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:45.820332] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:45.820338] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:45.820340] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:45.820346] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.820349] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.820350] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337645820298) [2024-09-26 16:00:45.820663] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.820673] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.820686] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.820789] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.820801] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.820807] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.821504] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:45.821515] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337645821502], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:45.821523] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337645821502], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:45.821540] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:45.821544] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:45.821650] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.822341] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.822354] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.822385] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.822402] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:45.836440] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.836449] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.851801] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:45.851918] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=53, refresh_score_time=27, wash_time=2) [2024-09-26 16:00:45.852566] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=377484, clean_num=31457) [2024-09-26 16:00:45.854445] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337645821502]}) [2024-09-26 16:00:45.866612] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337646796778, now=[mts=1727337644866989]) [2024-09-26 16:00:45.866629] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=17][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337646796778, now=[mts=1727337644866989], now0=[mts=1727337644866989], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:45.866637] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a3125a80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500763, assoc_session_id:3221500763, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337644866232, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:45.866654] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22f01f8, id:3221500763, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3125a80}) [2024-09-26 16:00:45.866662] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500763, *tx_desc={this:0x7f92a3125a80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500763, assoc_session_id:3221500763, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337644866232, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:45.866678] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=44] success to notify_need_writing_throttling True [2024-09-26 16:00:45.866669] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500763, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3125a80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500763, assoc_session_id:3221500763, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337644866232, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337644866966, use_das=false, nested_level=0, session={this:0x7f92a22f01f8, id:3221500763, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3125a80}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:45.866686] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:45.866691] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:45.866694] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:45.866687] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.866696] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:45.866699] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:45.866702] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:45.866707] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=1000072, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337646866778) [2024-09-26 16:00:45.866710] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:45.866717] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:45.866719] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:45.866720] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:45.866727] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:45.866749] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2169) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2] need check schema(new_schema_version=1, cached_tenant_schema_version=1, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = ? ORDER BY row_id, column_name) [2024-09-26 16:00:45.866771] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=5][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:45.866773] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:45.890226] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=5] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-09-26 16:00:45.890236] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=6] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=689) [2024-09-26 16:00:45.896726] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:45.896734] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.920315] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:45.920325] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337645920312) [2024-09-26 16:00:45.920331] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337645720310, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:45.920347] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223375156134, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:45.920353] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:45.920359] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:45.920363] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:45.920377] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:45.920383] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.920385] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:45.920387] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337645920334) [2024-09-26 16:00:45.926887] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:45.926896] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.930651] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337645821502]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:45.930659] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=8][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:45.955222] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337645821502]}) [2024-09-26 16:00:45.957114] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:45.957119] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:45.965479] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:45.979596] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=12] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:45.987415] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:45.987461] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.015646] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500768, support_ssl=false) [2024-09-26 16:00:46.015675] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221500768, sess.client_addr_="127.0.0.1:36352") [2024-09-26 16:00:46.015682] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500768, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:46.015730] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:46.015740] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2688][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=5] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:46.015788] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA1-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:46.015975] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA1-0-0] [lt=7][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:46.015983] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA1-0-0] [lt=6][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"^*TJTHJ{NS4L^w[@n*&}"}, ret=-4043) [2024-09-26 16:00:46.015990] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA1-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:46.015994] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA1-0-0] [lt=2][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:46.016002] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA1-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:46.016027] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA1-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500768, proxy_sessid:0}) [2024-09-26 16:00:46.016031] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA1-0-0] [lt=3] mark session id unused(sessid=3221500768) [2024-09-26 16:00:46.016034] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA1-0-0] [lt=2][errcode=0] server close connection(sessid=3221500768, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:46.016037] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA1-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500768, proxy_sessid=0) [2024-09-26 16:00:46.016040] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA1-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:46.016043] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA1-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500768, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:46.016063] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500768, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337646014896, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:46.016071] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500768, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:46.016074] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500768, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337646014896, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:46.016079] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500768, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:46.017518] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:46.017526] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.020505] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:46.020512] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:46.020519] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337646020503) [2024-09-26 16:00:46.020524] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337645920334, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:46.020529] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:46.020538] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:46.020540] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:46.020864] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:46.020871] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:46.020875] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:46.020878] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:46.020882] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:46.020884] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:46.020887] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:46.020890] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:46.020892] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:46.020893] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:46.020895] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:46.020898] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:46.020900] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:46.020902] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:46.020908] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:46.020910] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:46.020913] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:46.020915] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:46.020917] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:46.020920] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:46.020922] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:46.020928] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:46.020933] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:46.020935] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:46.020936] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:46.020942] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:46.020945] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:46.020947] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575A-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:46.020950] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:46.020952] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:46.020954] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:46.020957] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337646020787, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:46.020960] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:46.020962] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:46.020985] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:46.020989] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337646020988, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=454, wlock_time=8, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:46.020995] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:46.020997] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:46.021016] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223375257214, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:46.021021] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:46.021027] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:46.021031] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:46.021034] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:46.021039] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.021041] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.021043] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337646021001) [2024-09-26 16:00:46.023612] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185914-0-0] [lt=65][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:46.023625] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185914-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:46.023652] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185914-0-0] [lt=26][errcode=-4018] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:46.030249] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337645029517]) [2024-09-26 16:00:46.030259] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337645029517], now0=[mts=1727337645029517], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:46.030265] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337645028369, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:46.030281] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:46.030290] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337645028369, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:46.030297] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337645028369, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:46.030314] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:46.030318] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:46.030322] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:46.030326] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:46.030328] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:46.030332] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=8, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:46.030336] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] will sleep(sleep_us=8000, remain_us=20963996, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=8, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:46.038433] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:8, local_retry_times:8, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:46.038443] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:46.038445] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:46.038447] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:46.038452] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:46.038496] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:46.038502] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:46.047630] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:46.047635] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.047645] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:46.048078] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:46.048091] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:46.048095] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:46.048099] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:46.048102] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:46.048104] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:46.048107] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:46.048109] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:46.048111] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:46.048113] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:46.048115] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:46.048118] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:46.048122] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:46.048124] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:46.048131] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:46.048133] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:46.048136] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:46.048139] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:46.048141] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:46.048143] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:46.048146] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:46.048150] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:46.048155] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:46.048157] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:46.048159] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:46.048164] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:46.048168] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85887-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:46.048171] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:46.048174] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:46.048176] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:46.048181] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85887-0-0] [lt=5][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:46.048184] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337646048023, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:46.048187] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85887-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:46.048189] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85887-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:46.048207] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:46.048209] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85887-0-0] [lt=2][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:46.052775] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:46.052887] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=50, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:46.053525] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=408941, clean_num=31457) [2024-09-26 16:00:46.055293] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337645821502]}) [2024-09-26 16:00:46.058512] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=9] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:46.058786] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:46.058794] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:46.058798] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:46.058801] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:46.058804] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:46.058807] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:46.058809] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:46.058811] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:46.058812] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:46.058814] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:46.058816] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:46.058818] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:46.058820] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:46.058825] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:46.058830] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:46.058832] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:46.058834] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:46.058835] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:46.058837] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:46.058840] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:46.058842] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:46.058846] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:46.058851] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:46.058852] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:46.058854] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:46.058858] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:46.058861] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E68-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:46.058864] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2579][RSAsyncTask0][T0][YB42C0A8002D-0006230057584E68-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:46.058867] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:46.058869] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:46.058873] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:46.058875] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337646058746, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:46.058878] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:46.058923] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:46.068604] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=7][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4018, dropped:3, tid:2922}]) [2024-09-26 16:00:46.073425] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: SHARED_BLOCK_ReaderWriter_IO, mode: read, size: 4096.00, iops: 52.84, rt: 2690.47) [2024-09-26 16:00:46.073433] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:46.073465] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 13495.00) [2024-09-26 16:00:46.073468] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:46.073475] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:46.073479] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:46.077736] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:46.077746] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.085070] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:46.108175] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:46.108181] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.120834] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:46.120844] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337646120832) [2024-09-26 16:00:46.120849] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337646020526, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:46.120866] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223375357144, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:46.120871] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:46.120878] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:46.120882] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:46.120885] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:46.120890] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.120892] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.120894] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337646120852) [2024-09-26 16:00:46.138267] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:46.138275] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.156444] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337645821502]}) [2024-09-26 16:00:46.156461] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D8E2-0-0] [lt=9][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337646156421}) [2024-09-26 16:00:46.165499] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:46.168319] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.168328] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.198438] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.198447] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.199411] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:00:46.199419] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=7] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:46.199428] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:46.220975] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:46.220988] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337646220973) [2024-09-26 16:00:46.220992] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337646120852, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:46.221007] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223375456936, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:46.221012] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:46.221020] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:46.221024] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:46.221027] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:46.221032] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.221034] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.221036] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337646220995) [2024-09-26 16:00:46.228512] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:46.228519] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.244636] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:46.244647] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:46.244652] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:46.244655] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:46.244658] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:46.244661] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:46.244667] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=5][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:46.244670] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:46.244672] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:46.244684] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=12][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:46.244688] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=3][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:46.244690] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:46.244693] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:46.244695] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:46.244702] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:46.244704] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:46.244708] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:46.244712] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:46.244715] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:46.244718] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:46.244720] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:46.244726] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:46.244732] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:46.244734] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:46.244735] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:46.244742] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:46.244745] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:46.244748] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:46.244750] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:46.244753] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:46.244758] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:46.244761] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337646244569, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:46.244764] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:46.244766] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:46.244796] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=4][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:46.244798] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:46.244801] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=3][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:46.244804] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:46.244807] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:46.244810] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:46.244812] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485896-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:46.253767] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:46.253874] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=52, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:46.254488] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=440398, clean_num=31457) [2024-09-26 16:00:46.256619] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337645821502]}) [2024-09-26 16:00:46.258641] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.258647] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.288805] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.288818] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.308239] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337646236605, now=[mts=1727337645307820]) [2024-09-26 16:00:46.308253] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=13][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337646236605, now=[mts=1727337645307820], now0=[mts=1727337645307820], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:46.308260] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926915bbc0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337645307657, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:46.308278] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915bbc0}) [2024-09-26 16:00:46.308287] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926915bbc0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337645307657, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:46.308294] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926915bbc0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337645307657, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337644306849, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915bbc0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:46.308312] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:46.308317] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:46.308320] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:46.308323] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:46.308326] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:46.308330] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:46.308335] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:46.308342] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:46.308344] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:46.308346] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:46.308351] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:46.308354] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2001505) [2024-09-26 16:00:46.308357] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:46.308362] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:46.308365] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:46.308367] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:46.308370] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:46.308378] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=5][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:46.308432] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:46.308436] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:46.308440] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=3][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:46.308442] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:46.308447] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=4][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:46.308450] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:46.308571] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7D-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:46.308582] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:46.308584] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:46.308602] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:46.308607] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:46.308897] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:46.308904] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=7][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:46.319523] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25] success to notify_need_writing_throttling True [2024-09-26 16:00:46.319534] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.321502] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:46.321512] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337646321498) [2024-09-26 16:00:46.321517] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337646220995, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:46.321534] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223375557952, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:46.321540] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:46.321547] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:46.321552] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:46.321554] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:46.321560] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.321562] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.321563] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337646321520) [2024-09-26 16:00:46.322619] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.322629] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337646322617], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:46.322636] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337646322617], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:46.322640] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=6164, get_gts_with_stc_cnt=14582, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-09-26 16:00:46.349830] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:46.349837] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.356649] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337646322617]}) [2024-09-26 16:00:46.363551] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D3CDF-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:46.366421] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:46.368445] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1727337646368445, bytes: 14654086, bw: 0.003375 MB/s, add_ts: 1000192, add_bytes: 3540 [2024-09-26 16:00:46.380113] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.380123] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.387908] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:46.387925] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=10] current monitor number(seq_=-1) [2024-09-26 16:00:46.400902] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4F-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:46.400917] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4F-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:46.400924] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4F-0-0] [lt=5][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:46.400927] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4F-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:46.400930] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4F-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:46.400934] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4F-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:46.400935] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4F-0-0] [lt=1][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:46.400938] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4F-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:46.400953] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4F-0-0] [lt=13][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:46.400956] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4F-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:46.400997] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F4F-0-0] [lt=0] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:46.410399] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.410409] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.422114] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:46.422128] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337646422110) [2024-09-26 16:00:46.422134] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337646321520, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:46.422152] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223375658606, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:46.422160] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:46.422167] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:46.422172] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:46.422177] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:46.422184] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.422186] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.422188] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337646422138) [2024-09-26 16:00:46.441634] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:46.441652] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.455433] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:46.455548] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=52, refresh_score_time=27, wash_time=1) [2024-09-26 16:00:46.456147] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=471855, clean_num=31457) [2024-09-26 16:00:46.457308] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337646322617]}) [2024-09-26 16:00:46.457310] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.457326] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.457331] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.457340] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.457346] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.457351] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.458232] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=20] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.458233] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.458236] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.458236] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.458238] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.458239] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.458270] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.458277] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.458281] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.459198] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.459208] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.459212] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.459221] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.459224] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.459226] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.459227] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.459230] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.459232] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.460190] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.460196] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.460199] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.460205] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.460207] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.460208] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.460213] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.460215] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.460217] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.461187] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.461192] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.461195] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.461200] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.461201] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.461203] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.461268] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.461276] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.461280] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.461762] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.461771] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.461775] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.461779] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.461785] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.461788] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:46.461825] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.463091] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.463101] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.463108] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.464081] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.464093] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.464096] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.465048] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.465069] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.465076] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.466045] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.466051] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.466052] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.467009] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.467026] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.467034] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.467986] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.468012] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.468044] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.468982] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.468988] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.468996] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.469974] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.469975] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.469987] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.470940] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.470945] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.470946] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.471896] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.471907] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.471910] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:46.471916] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.471932] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.472881] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.472884] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.472885] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.473854] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.473891] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.473892] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.474827] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.474828] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.474830] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.475813] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.475818] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.475815] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.476787] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.476793] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.476832] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.477760] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.477760] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.477774] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.478731] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.478742] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.478743] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.479706] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.479706] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.479711] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.480685] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.481691] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.481698] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.481704] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.482661] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.482663] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.482664] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.483624] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.483626] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.483636] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.484603] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.484611] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.484615] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.485577] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.485582] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.485585] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.486539] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.486555] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.486556] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.487539] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.487545] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.487545] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.488502] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.488525] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.488527] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.489501] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.489504] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.489508] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.490056] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:46.502214] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.502236] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.521827] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185915-0-0] [lt=0][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:46.521860] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185915-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:46.521873] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185915-0-0] [lt=12][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:46.521880] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185915-0-0] [lt=6][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:46.522693] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:46.522710] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337646522690) [2024-09-26 16:00:46.522716] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337646422137, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:46.522721] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:46.522739] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:46.522743] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:46.523060] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575B-0-0] [lt=6] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:46.523071] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575B-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:46.523099] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575B-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:46.523133] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337646523132, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=406, wlock_time=23, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:46.523145] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:46.523162] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223375759223, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:46.523167] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:46.523174] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:46.523178] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:46.523180] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:46.523186] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.523189] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.523190] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337646523150) [2024-09-26 16:00:46.532485] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:46.532493] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.543214] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6480, mem_used=12478464) [2024-09-26 16:00:46.557887] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337646322617]}) [2024-09-26 16:00:46.562792] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.562799] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.563704] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=10] gc stale ls task succ [2024-09-26 16:00:46.566950] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=41] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=770672, replace_num=15728) [2024-09-26 16:00:46.566960] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:46.593025] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.593031] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.596953] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=5][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337646322617]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:46.596964] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=11][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:46.601804] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.601804] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.602792] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.602803] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.623321] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:46.623341] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337646623303) [2024-09-26 16:00:46.623346] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337646522719, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:46.623346] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.623347] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:46.623351] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.623362] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223375859809, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:46.623369] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:46.623384] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:46.623391] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:46.623393] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:46.623398] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.623401] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.623403] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337646623348) [2024-09-26 16:00:46.653558] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.653567] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.656628] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:46.656742] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:46.657369] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=503312, clean_num=31457) [2024-09-26 16:00:46.658478] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337646322617]}) [2024-09-26 16:00:46.661433] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=4] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:46.671224] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4184, dropped:490, tid:2850}, {errcode:-5019, dropped:49, tid:2634}]) [2024-09-26 16:00:46.683901] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.683909] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.701462] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:46.714165] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:46.714181] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.723905] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:46.723929] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337646723901) [2024-09-26 16:00:46.723935] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337646623348, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:46.723951] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223375960401, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:46.723957] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:46.723964] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:46.723970] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:46.723972] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:46.723978] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.723981] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.723983] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337646723938) [2024-09-26 16:00:46.744440] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:46.744450] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.759069] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337646322617]}) [2024-09-26 16:00:46.767839] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=4] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:46.775662] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21] success to notify_need_writing_throttling True [2024-09-26 16:00:46.775670] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.805965] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:46.805972] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.812907] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [2915][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=8] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-09-26 16:00:46.814738] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337646814737, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008075, add_bytes: 0 [2024-09-26 16:00:46.821576] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.821582] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.821591] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.821589] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.821595] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.821596] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.821602] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.821603] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.821609] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.821615] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.821622] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:46.823666] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:46.823676] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337646823663], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:46.823683] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337646823663], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:46.823687] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:46.823691] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:46.824492] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:46.824501] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337646824490) [2024-09-26 16:00:46.824505] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337646723938, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:46.824519] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223376060040, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:46.824525] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:46.824531] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:46.824536] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:46.824541] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:46.824546] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.824548] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.824550] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337646824507) [2024-09-26 16:00:46.836210] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:46.836216] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.857824] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:46.857925] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=49, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:46.858515] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=534769, clean_num=31457) [2024-09-26 16:00:46.859651] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337646823663]}) [2024-09-26 16:00:46.866499] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:46.866505] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.867469] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337646796778, now=[mts=1727337645866769]) [2024-09-26 16:00:46.867477] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337646796778, now=[mts=1727337645866769], now0=[mts=1727337645866769], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:46.867483] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a3125a80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500763, assoc_session_id:3221500763, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337645866307, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:46.867500] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22f01f8, id:3221500763, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3125a80}) [2024-09-26 16:00:46.867507] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500763, *tx_desc={this:0x7f92a3125a80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500763, assoc_session_id:3221500763, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337645866307, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:46.867515] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500763, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3125a80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500763, assoc_session_id:3221500763, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337645866307, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337644866966, use_das=false, nested_level=0, session={this:0x7f92a22f01f8, id:3221500763, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3125a80}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:46.867532] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:46.867538] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:46.867541] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:46.867543] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:46.867546] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:46.867550] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:46.867555] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4] already timeout, do not need sleep(sleep_us=-777, remain_us=-777, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, timeout_timestamp=1727337646866778) [2024-09-26 16:00:46.867559] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:46.867566] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:46.867568] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:46.867569] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:46.867575] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:46.867613] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1764) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4012] get gts timeout(ret=-4012, expire_ts=1727337646796778, current_time=1727337646866585) [2024-09-26 16:00:46.867618] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=5][errcode=-4012] acquire global snapshot fail(ret=-4012, gts_ahead=0, expire_ts=1727337646796778, now=[mts=1727337646867612], now0=[mts=1727337646867612], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:46.867621] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4012] acquire global snapshot fail(ret=-4012, tx={this:0x7f92a3125a80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500763, assoc_session_id:3221500763, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337646866585, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:46.867639] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=18][errcode=-4012] fail to get snapshot(ret=-4012, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22f01f8, id:3221500763, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3125a80}) [2024-09-26 16:00:46.867644] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=5][errcode=-4012] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4012, session_id=3221500763, *tx_desc={this:0x7f92a3125a80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500763, assoc_session_id:3221500763, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337646866585, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:46.867650] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=6] start stmt(ret=-4012, auto_commit=true, session_id=3221500763, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3125a80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500763, assoc_session_id:3221500763, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337646866585, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337644866966, use_das=false, nested_level=0, session={this:0x7f92a22f01f8, id:3221500763, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3125a80}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:46.867661] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=11][errcode=-4012] fail to start stmt(ret=-4012, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:46.867664] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4012] fail start stmt(ret=-4012) [2024-09-26 16:00:46.867665] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1][errcode=-4012] execute plan failed(ret=-4012) [2024-09-26 16:00:46.867668] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4012] open result set failed(ret=-4012) [2024-09-26 16:00:46.867669] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:46.867671] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:46.867674] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:46.867677] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:46.867678] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:46.867682] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=4][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-09-26 16:00:46.867686] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:46.867688] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=1] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000722) [2024-09-26 16:00:46.867691] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885085-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:46.867694] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:46.867697] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:46.867699] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:46.867702] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8620, start=1727337644866963, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:46.867706] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:00:46.867709] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:46.867737] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:46.867739] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:46.867744] WDIAG [SHARE] get_snapshot_gc_scn (ob_global_stat_proxy.cpp:164) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:46.867746] WDIAG [STORAGE] get_global_info (ob_tenant_freeze_info_mgr.cpp:811) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to get global info(ret=-4012, tenant_id=1) [2024-09-26 16:00:46.867749] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:954) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] failed to get global info(ret=-4012) [2024-09-26 16:00:46.867751] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1008) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to try update info(tmp_ret=-4012, tmp_ret="OB_TIMEOUT") [2024-09-26 16:00:46.896804] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.896813] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.925135] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:46.925147] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337646925132) [2024-09-26 16:00:46.925152] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337646824507, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:46.925169] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223376160630, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:46.925174] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:46.925181] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:46.925185] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:46.925188] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:46.925193] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.925195] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:46.925196] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337646925155) [2024-09-26 16:00:46.927120] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.927129] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.958341] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:46.958347] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:46.960265] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337646823663]}) [2024-09-26 16:00:46.968062] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:46.980767] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] tx gc loop thread is running(MTL_ID()=1) [2024-09-26 16:00:46.980776] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] try gc retain ctx [2024-09-26 16:00:46.981160] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-09-26 16:00:46.981187] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:46.988614] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:46.988621] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.016404] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500772, support_ssl=false) [2024-09-26 16:00:47.016440] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221500772, sess.client_addr_="127.0.0.1:36354") [2024-09-26 16:00:47.016461] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f9264404030, session_id:3221500772, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:47.016484] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:47.016495] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:47.016604] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA1-0-0] [lt=8] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:47.016854] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA1-0-0] [lt=7][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:47.016863] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA1-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"#tB*L~Vr[PatE;9HBPKg"}, ret=-4043) [2024-09-26 16:00:47.016870] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA1-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:47.016875] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA1-0-0] [lt=4][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:47.016884] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA1-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:47.016909] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA1-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500772, proxy_sessid:0}) [2024-09-26 16:00:47.016913] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA1-0-0] [lt=4] mark session id unused(sessid=3221500772) [2024-09-26 16:00:47.016918] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA1-0-0] [lt=3][errcode=0] server close connection(sessid=3221500772, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:47.016921] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA1-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500772, proxy_sessid=0) [2024-09-26 16:00:47.016924] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA1-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:47.016926] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA1-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500772, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:47.016960] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f9264404030, session_id:3221500772, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337647016341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:47.016969] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] kill and revert session(conn.sessid_=3221500772, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:47.016982] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f9264404030, session_id:3221500772, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337647016341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:47.016987] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500772, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:47.018868] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:47.018877] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.024183] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:47.024196] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-09-26 16:00:47.024201] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:47.024206] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-09-26 16:00:47.024211] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:47.024214] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:47.024219] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-09-26 16:00:47.024222] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:47.024224] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:47.024227] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=3][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:47.024230] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=3][errcode=-5019] resolve table failed(ret=-5019) [2024-09-26 16:00:47.024233] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:2822) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=2][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-09-26 16:00:47.024236] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2676) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=2][errcode=-5019] resolve joined table failed(ret=-5019) [2024-09-26 16:00:47.024238] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:47.024241] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:47.024244] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:47.024247] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:47.024255] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:47.024258] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:47.024262] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:47.024264] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:47.024268] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-09-26 16:00:47.024271] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:47.024274] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:47.024281] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:47.024289] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:47.024292] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:47.024294] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:47.024301] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:47.024306] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA1-0-0] [lt=4][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:47.024309] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [3024][BlackListServic][T0][YB42C0A8002D-0006230056284FA1-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-09-26 16:00:47.024313] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:47.024316] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:47.024319] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:47.024322] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337647024056, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:47.024327] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:47.024330] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:222) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:47.024357] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3] ls blacklist refresh finish(cost_time=631) [2024-09-26 16:00:47.025720] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:47.025728] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:47.025736] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337647025717) [2024-09-26 16:00:47.025740] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337646925155, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:47.025746] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:47.025754] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:47.025757] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:47.026037] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:47.026044] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:47.026047] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:47.026052] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:47.026055] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:47.026057] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:47.026060] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:47.026062] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:47.026064] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:47.026065] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:47.026067] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:47.026069] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:47.026071] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:47.026073] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:47.026077] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:47.026079] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:47.026081] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:47.026084] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:47.026086] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:47.026088] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:47.026090] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:47.026095] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:47.026099] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:47.026101] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:47.026102] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:47.026107] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:47.026109] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:47.026111] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575C-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:47.026115] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:47.026118] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:47.026120] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:47.026122] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337647025982, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:47.026125] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:47.026127] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:47.026149] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:47.026152] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337647026151, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=400, wlock_time=8, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:47.026158] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:47.026161] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:47.026177] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223376262218, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:47.026181] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:47.026187] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:47.026191] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:47.026193] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:47.026198] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.026200] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.026202] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337647026164) [2024-09-26 16:00:47.026318] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185916-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:47.026327] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185916-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:47.026337] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185916-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:47.026342] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185916-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:47.040359] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337646038495]) [2024-09-26 16:00:47.040370] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337646038495], now0=[mts=1727337646038495], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:47.040389] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337646038430, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:47.040407] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:47.040414] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337646038430, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:47.040422] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337646038430, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:47.040438] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:47.040442] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:47.040445] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:47.040450] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:47.040452] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:47.040455] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=9, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:47.040470] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14] will sleep(sleep_us=9000, remain_us=19953862, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=9, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:47.048507] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:47.048518] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:47.048522] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:47.048525] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:47.048528] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:47.048530] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:47.048533] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:47.048535] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:47.048537] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:47.048538] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:47.048540] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:47.048541] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:47.048543] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:47.048564] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=20][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:47.048568] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:47.048570] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:47.048572] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:47.048574] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:47.048575] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:47.048578] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:47.048579] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:47.048584] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:47.048592] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=7][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:47.048594] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:47.048595] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:47.048601] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:47.048603] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:47.048606] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:47.048608] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:47.048610] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:47.048612] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:47.048614] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85888-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337647048459, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:47.048617] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85888-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:47.048619] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85888-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:47.048638] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85888-0-0] [lt=5][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:47.048641] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85888-0-0] [lt=3][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:47.049150] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:47.049160] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.049170] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:47.050103] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:9, local_retry_times:9, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:47.050126] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=19][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:47.050130] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:47.050132] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:47.050138] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:47.050168] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:47.050183] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:47.060002] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:47.060103] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:47.060448] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337646823663]}) [2024-09-26 16:00:47.060739] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=566226, clean_num=31457) [2024-09-26 16:00:47.060754] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=6] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:47.061002] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:47.061011] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:47.061014] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:47.061029] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:47.061033] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:47.061035] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:47.061038] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:47.061040] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:47.061042] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:47.061044] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:47.061046] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:47.061048] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:47.061050] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:47.061052] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:47.061055] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:47.061058] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:47.061060] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:47.061062] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:47.061064] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:47.061066] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:47.061068] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:47.061072] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:47.061075] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:47.061077] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:47.061079] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:47.061083] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:47.061085] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6A-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:47.061091] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2579][RSAsyncTask0][T0][YB42C0A8002D-0006230057584E6A-0-0] [lt=5][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:47.061093] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:47.061095] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:47.061097] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:47.061099] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337647060964, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:47.061102] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:47.061145] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:47.078493] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10796.00) [2024-09-26 16:00:47.078503] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:47.078514] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:47.078517] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:47.079366] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:47.079377] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.085291] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=8] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:47.110675] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:47.110684] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.126297] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:47.126308] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337647126295) [2024-09-26 16:00:47.126313] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337647025743, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:47.126367] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223376362777, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:47.126375] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:47.126382] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:47.126387] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:47.126390] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:47.126396] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.126398] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.126400] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337647126316) [2024-09-26 16:00:47.140949] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:47.140971] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.143825] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:221) [2596][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=47] [ASYNC TASK QUEUE](queue_size=0) [2024-09-26 16:00:47.150743] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=33] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:47.160480] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337646823663]}) [2024-09-26 16:00:47.160501] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D8EE-0-0] [lt=6][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337647160473}) [2024-09-26 16:00:47.168342] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:47.171252] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:47.171267] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.201506] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:47.201513] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.205633] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:47.205645] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:47.205661] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=16] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:47.226330] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:47.226354] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337647226327) [2024-09-26 16:00:47.226360] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337647126316, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:47.226375] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223376462371, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:47.226381] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:47.226388] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:47.226393] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:47.226395] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:47.226401] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.226404] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.226405] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337647226363) [2024-09-26 16:00:47.231559] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:47.231566] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.245209] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=5] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:47.245220] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:47.245224] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:47.245228] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:47.245231] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:47.245233] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:47.245237] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:47.245240] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:47.245242] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:47.245244] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:47.245246] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:47.245251] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:47.245253] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:47.245255] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:47.245275] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=15][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:47.245278] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:47.245281] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:47.245284] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:47.245287] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:47.245290] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:47.245293] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:47.245298] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:47.245304] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:47.245306] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:47.245308] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:47.245313] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:47.245317] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:47.245320] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:47.245325] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:47.245328] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:47.245330] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:47.245333] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337647245144, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:47.245344] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:47.245348] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:47.245377] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=3][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:47.245380] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=3][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:47.245389] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=9][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:47.245392] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=3][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:47.245395] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=3][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:47.245399] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=3][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:47.245401] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485897-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:47.260532] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=21] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337646823663]}) [2024-09-26 16:00:47.261241] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:47.261342] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=48, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:47.261964] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=597683, clean_num=31457) [2024-09-26 16:00:47.262065] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:47.262069] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.292346] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:47.292355] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.308723] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=7][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337648238611, now=[mts=1727337646308895]) [2024-09-26 16:00:47.308737] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=12][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337648238611, now=[mts=1727337646308895], now0=[mts=1727337646308895], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:47.308744] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926915d960, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337646307903, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:47.308762] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915d960}) [2024-09-26 16:00:47.308771] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=9][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926915d960, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337646307903, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:47.308780] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=9] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926915d960, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337646307903, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337646308866, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915d960}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:47.308797] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:47.308802] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:47.308805] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:47.308808] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:47.308811] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:47.308815] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:47.308821] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=999791, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337648308611) [2024-09-26 16:00:47.308825] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:47.308833] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:47.308835] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:47.308836] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:47.308843] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:47.308889] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:47.308894] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:47.322470] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:47.322479] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.324009] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.324019] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337647324007], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:47.324028] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337647324007], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:47.324032] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337647324007]}) [2024-09-26 16:00:47.326370] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:47.326379] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337647326368) [2024-09-26 16:00:47.326383] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337647226363, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:47.326404] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223376562631, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:47.326411] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:47.326418] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:47.326424] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:47.326427] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:47.326433] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.326435] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.326438] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337647326387) [2024-09-26 16:00:47.352569] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:47.352577] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.360972] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337647324007]}) [2024-09-26 16:00:47.368469] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:47.372504] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D3D27-0-0] [lt=8][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:47.375347] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337647375346, bytes: 14657908, bw: 0.003620 MB/s, add_ts: 1006901, add_bytes: 3822 [2024-09-26 16:00:47.383162] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:47.383168] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.388042] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:47.388057] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=7] current monitor number(seq_=-1) [2024-09-26 16:00:47.413491] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:47.413501] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.424788] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] ====== tenant freeze timer task ====== [2024-09-26 16:00:47.424800] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:47.424833] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337645627740, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337645627695, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337645627760, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:47.424860] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:47.426442] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:47.426457] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337647426439) [2024-09-26 16:00:47.426463] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337647326386, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:47.426498] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223376661977, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:47.426515] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:47.426523] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:47.426529] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:47.426532] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:47.426539] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.426541] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.426543] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337647426466) [2024-09-26 16:00:47.426840] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DC-0-0] [lt=6] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:47.426847] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DC-0-0] [lt=5] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:47.426853] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DC-0-0] [lt=4] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:47.426856] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DC-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:47.426859] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DC-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:47.426872] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DC-0-0] [lt=12][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:47.426875] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DC-0-0] [lt=3][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:47.426885] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DC-0-0] [lt=10] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:47.426890] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DC-0-0] [lt=5][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:47.426892] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DC-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:47.426895] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DC-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:47.426938] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851DC-0-0] [lt=6] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:47.443724] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:47.443734] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.457832] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.457843] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.457849] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.458367] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.458378] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.458383] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.458421] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.458431] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.458436] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.459138] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.459143] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.459146] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.459193] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.459196] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.459202] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.459686] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.459692] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.459694] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.460309] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.460315] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.460317] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.460753] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.460759] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.460761] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.460853] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.460857] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.460860] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.461338] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.461345] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.461350] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.461357] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337647324007]}) [2024-09-26 16:00:47.461417] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.461423] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.461425] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.462040] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.462045] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.462048] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.462073] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.462076] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.462078] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.462206] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:47.462317] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=17] Wash time detail, (compute_wash_size_time=60, refresh_score_time=16, wash_time=2) [2024-09-26 16:00:47.462608] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.462611] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.462613] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.462636] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.462639] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.462641] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.462932] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=629140, clean_num=31457) [2024-09-26 16:00:47.463272] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.463276] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.463279] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:47.463296] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.463855] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.463862] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.464500] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.464526] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.465130] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.465377] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.465776] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.466001] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.466348] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.467156] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.467176] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.467901] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.467913] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.468466] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.468470] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.469107] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.469357] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.469794] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.469882] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.470356] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.470555] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.470959] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.471159] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.471485] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.472041] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.472317] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.472569] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.473057] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.473175] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.473991] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.474095] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:47.474100] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.474164] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.474555] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.474743] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.475121] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.475399] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.475947] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.475948] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.476540] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.476923] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.477118] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.477582] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.477680] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.478141] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.478376] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.478725] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.479000] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.479243] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.479727] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.479921] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.480438] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.480465] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.480991] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.480993] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.481531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.481617] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.482103] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.482273] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.482937] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.483104] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.483478] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.483627] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.484261] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.484520] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.485374] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.485412] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.485938] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.486131] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.486534] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.486731] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.487195] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.487307] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.487789] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.487947] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.488356] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.488580] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.489200] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.489235] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.489787] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.489793] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.490407] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.490407] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.491521] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.491528] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:47.504265] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:47.504272] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.523334] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185917-0-0] [lt=38][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:47.523346] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185917-0-0] [lt=11][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:47.523355] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185917-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:47.523359] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185917-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:47.526498] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:47.526511] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337647526495) [2024-09-26 16:00:47.526516] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337647426466, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:47.526521] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:47.526530] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:47.526533] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:47.526903] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575D-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:47.526916] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575D-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:47.526920] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575D-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:47.526924] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575D-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:47.526927] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575D-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:47.526929] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575D-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:47.526933] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575D-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:47.526937] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575D-0-0] [lt=3][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:47.526958] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575D-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:47.526992] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337647526992, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=466, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:47.527002] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:47.527018] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223376763188, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:47.527024] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:47.527030] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:47.527035] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:47.527038] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:47.527044] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.527047] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.527048] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337647527006) [2024-09-26 16:00:47.534760] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:47.534768] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.543871] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6490, mem_used=12478464) [2024-09-26 16:00:47.561482] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337647324007]}) [2024-09-26 16:00:47.563749] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=5] gc stale ls task succ [2024-09-26 16:00:47.564957] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:47.564962] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.568668] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:47.595077] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:47.595083] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.601044] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.601056] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.601344] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.602533] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.625411] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:47.625419] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.627360] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:47.631257] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:47.631267] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337647631254) [2024-09-26 16:00:47.631272] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337647526519, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:47.631291] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223376867208, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:47.631297] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:47.631305] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:47.631310] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:47.631313] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:47.631319] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.631322] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.631324] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337647631275) [2024-09-26 16:00:47.634943] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:45) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2] dump dtl interm result cost(us)(dump_cost=13665, ret=0, interm count=0, dump count=344) [2024-09-26 16:00:47.648395] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:60) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11] clear dtl interm result cost(us)(clear_cost=13439, ret=0, expire_keys_.count()=0, dump count=344, interm count=0, clean count=0) [2024-09-26 16:00:47.648819] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD4-0-0] [lt=14] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:47.648830] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD4-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:47.648863] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD4-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:47.655462] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:47.655470] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.661497] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:47.661514] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337647324007]}) [2024-09-26 16:00:47.663613] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:47.663748] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=58, refresh_score_time=15, wash_time=20) [2024-09-26 16:00:47.664337] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=660597, clean_num=31457) [2024-09-26 16:00:47.685542] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:47.685548] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.687414] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=24][errcode=-4018] try free send task failed(ret=-4018) [2024-09-26 16:00:47.705475] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:47.715669] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:47.715688] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.731425] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:47.731441] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337647731422) [2024-09-26 16:00:47.731448] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337647631275, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:47.731477] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223376967254, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:47.731485] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:47.731493] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:47.731499] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:47.731501] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:47.731506] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.731509] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.731511] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337647731451) [2024-09-26 16:00:47.746492] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:47.746501] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.762143] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337647324007]}) [2024-09-26 16:00:47.768972] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=18] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:47.776767] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:47.776773] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.807044] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:47.807050] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.821490] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.821502] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.821686] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.821725] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.821735] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.821740] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.821747] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.821754] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.821766] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.821772] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.821776] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:47.821824] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=17] PNIO [ratelimit] time: 1727337647821823, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007086, add_bytes: 0 [2024-09-26 16:00:47.824504] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:47.824515] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337647824502], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:47.824522] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337647824502], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:47.824526] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:47.824530] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:47.831495] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:47.831505] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337647831492) [2024-09-26 16:00:47.831510] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337647731451, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:47.831528] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223377067671, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:47.831533] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:47.831541] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:47.831545] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:47.831547] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:47.831553] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.831555] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.831557] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337647831513) [2024-09-26 16:00:47.837327] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:47.837333] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.862301] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337647824502]}) [2024-09-26 16:00:47.864579] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:47.864692] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=48, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:47.865293] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=692054, clean_num=31457) [2024-09-26 16:00:47.867376] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:47.867381] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.897494] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:47.897500] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.927934] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:47.927939] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.932065] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:47.932074] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337647932063) [2024-09-26 16:00:47.932079] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337647831513, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:47.932095] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223377167189, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:47.932100] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:47.932107] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:47.932114] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:47.932117] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:47.932122] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.932124] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:47.932125] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337647932082) [2024-09-26 16:00:47.958051] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:47.958056] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:47.962469] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337647824502]}) [2024-09-26 16:00:47.969288] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:47.980196] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:47.988394] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:47.988399] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.018648] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:48.018655] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.018680] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221500776, support_ssl=false) [2024-09-26 16:00:48.018705] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] sm conn init succ(conn.sessid_=3221500776, sess.client_addr_="127.0.0.1:36356") [2024-09-26 16:00:48.018722] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500776, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:48.018887] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:48.018898] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2688][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=5] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:48.018950] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA0-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:48.019124] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA0-0-0] [lt=4][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:48.019129] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA0-0-0] [lt=4][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"hpw'Xh"m#bH|Z0|CwQlM"}, ret=-4043) [2024-09-26 16:00:48.019134] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA0-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:48.019138] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA0-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:48.019145] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA0-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:48.019167] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA0-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500776, proxy_sessid:0}) [2024-09-26 16:00:48.019170] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA0-0-0] [lt=3] mark session id unused(sessid=3221500776) [2024-09-26 16:00:48.019174] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA0-0-0] [lt=2][errcode=0] server close connection(sessid=3221500776, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:48.019177] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA0-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500776, proxy_sessid=0) [2024-09-26 16:00:48.019179] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA0-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:48.019181] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA0-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500776, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:48.019215] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500776, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337648018644, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:48.019223] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500776, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:48.019225] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=2] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500776, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337648018644, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:48.019241] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] connection close(sessid=3221500776, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:48.026318] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185918-0-0] [lt=40][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:48.026331] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185918-0-0] [lt=13][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:48.026341] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185918-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:48.026349] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185918-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:48.032590] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:48.032618] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:48.032637] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337648032588) [2024-09-26 16:00:48.032641] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337647932082, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:48.032647] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:48.032669] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:48.032674] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:48.032976] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575E-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:48.033007] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788575E-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:48.033038] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337648033037, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=384, wlock_time=23, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:48.033046] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:48.033062] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223377268145, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:48.033067] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:48.033073] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:48.033077] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:48.033079] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:48.033085] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.033087] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.033089] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337648033050) [2024-09-26 16:00:48.049121] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85889-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:48.049133] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85889-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:48.049159] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85889-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:48.049196] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:48.049200] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.049209] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:48.051228] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337647050168]) [2024-09-26 16:00:48.051237] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337647050168], now0=[mts=1727337647050168], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:48.051244] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337647050118, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:48.051261] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:48.051268] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337647050118, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:48.051276] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337647050118, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:48.051292] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:48.051297] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:48.051300] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:48.051303] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:48.051305] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:48.051310] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=10, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:48.051314] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=10000, remain_us=18943018, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=10, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:48.061250] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=3] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:48.061501] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6C-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:48.061507] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6C-0-0] [lt=5][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:48.061525] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6C-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:48.061980] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:10, local_retry_times:10, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:48.061998] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:48.062003] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:48.062005] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:48.062011] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:48.062044] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:48.062049] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:48.062963] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337647824502]}) [2024-09-26 16:00:48.065546] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:48.065644] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:48.066251] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=723511, clean_num=31457) [2024-09-26 16:00:48.075630] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:169, tid:3024}]) [2024-09-26 16:00:48.079314] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=26] success to notify_need_writing_throttling True [2024-09-26 16:00:48.079321] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.082099] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 12140.00) [2024-09-26 16:00:48.082110] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:48.082121] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:48.082125] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:48.085325] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:48.109418] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:48.109425] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.132897] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:48.132906] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337648132894) [2024-09-26 16:00:48.132911] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337648032644, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:48.132932] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223377368761, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:48.132937] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:48.132944] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:48.132948] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:48.132951] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:48.132958] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.132961] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.132962] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337648132914) [2024-09-26 16:00:48.139469] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:48.139474] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.151876] INFO [STORAGE] runTimerTask (ob_ddl_heart_beat_task.cpp:49) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=0] send to rs all task status succeed [2024-09-26 16:00:48.163052] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337647824502]}) [2024-09-26 16:00:48.163083] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D8FA-0-0] [lt=8][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337648163046}) [2024-09-26 16:00:48.169419] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:48.169545] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:48.169550] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.199646] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:00:48.199655] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=8] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:48.199664] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:48.199788] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:48.199794] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.229961] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:48.229967] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.233042] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:48.233056] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337648233040) [2024-09-26 16:00:48.233061] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337648132914, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:48.233076] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223377469481, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:48.233081] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:48.233088] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:48.233093] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:48.233095] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:48.233100] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.233102] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.233104] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337648233064) [2024-09-26 16:00:48.245831] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:48.245843] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:48.245847] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:48.245851] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:48.245855] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:48.245858] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:48.245865] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=5][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:48.245867] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:48.245869] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:48.245871] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:48.245873] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:48.245875] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:48.245879] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:48.245881] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:48.245888] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:48.245891] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:48.245894] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:48.245896] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:48.245899] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:48.245902] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:48.245907] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:48.245912] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:48.245918] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:48.245921] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:48.245922] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:48.245928] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:48.245932] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:48.245934] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:48.245936] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:48.245938] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:48.245941] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:48.245944] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337648245762, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:48.245947] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:48.245950] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:48.245977] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=3][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:48.245980] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:48.245983] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:48.245985] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:48.245988] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:48.245991] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:48.245993] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485898-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:48.260243] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:48.260249] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.263244] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=26] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337647824502]}) [2024-09-26 16:00:48.266891] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:48.266989] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=48, refresh_score_time=13, wash_time=2) [2024-09-26 16:00:48.267624] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=754968, clean_num=31457) [2024-09-26 16:00:48.290374] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:48.290380] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.299328] INFO [STORAGE.BLKMGR] mark_tenant_blocks (ob_block_manager.cpp:1090) [2632][BlkMgr][T1][Y0-0000000000000000-0-0] [lt=7] disk usage exceeds threshold, skip marking(io_device_->get_free_block_count()=0, super_block_.get_total_macro_block_count()=10240) [2024-09-26 16:00:48.299339] WDIAG [STORAGE.BLKMGR] mark_macro_blocks (ob_block_manager.cpp:1029) [2632][BlkMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] fail to mark tenant blocks(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.299350] INFO [STORAGE.BLKMGR] do_sweep (ob_block_manager.cpp:923) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=3] block manager free block(macro_id=[1091](ver=0,mode=0,seq=824626), io_fd={first_id:824626, second_id:1091, device_handle:null}) [2024-09-26 16:00:48.299356] INFO [STORAGE.BLKMGR] do_sweep (ob_block_manager.cpp:923) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=5] block manager free block(macro_id=[1812](ver=0,mode=0,seq=824627), io_fd={first_id:824627, second_id:1812, device_handle:null}) [2024-09-26 16:00:48.299781] INFO [STORAGE.BLKMGR] mark_and_sweep (ob_block_manager.cpp:998) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=2] finish once mark and sweep(ret=0, alloc_num=2, marker_status={total_block_count:10240, reserved_block_count:2, linked_block_count:0, tmp_file_count:0, data_block_count:0, shared_data_block_count:0, index_block_count:0, ids_block_count:0, disk_block_count:0, bloomfiter_count:0, hold_count:10234, pending_free_count:0, free_count:2, shared_meta_block_count:0, mark_cost_time:407, sweep_cost_time:15, start_time:"2024-09-26 16:00:48.298936", last_end_time:"2024-09-26 16:00:48.299358", mark_finished:false, hold_info:nothing}, map_cnt=10236) [2024-09-26 16:00:48.308285] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=7][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337648238611, now=[mts=1727337647308887]) [2024-09-26 16:00:48.308317] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=32][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337648238611, now=[mts=1727337647308887], now0=[mts=1727337647308887], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:48.308324] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926915d960, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337647308073, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:48.308339] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=14][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915d960}) [2024-09-26 16:00:48.308347] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926915d960, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337647308073, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:48.308354] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926915d960, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337647308073, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337646308866, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915d960}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:48.308370] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:48.308374] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:48.308377] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:48.308379] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:48.308393] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:48.308398] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:48.308402] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3] will sleep(sleep_us=209, remain_us=209, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4283, timeout_timestamp=1727337648308611) [2024-09-26 16:00:48.309075] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:48.309084] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=5][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:48.309086] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:48.309088] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:48.309092] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:48.309130] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1764) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4012] get gts timeout(ret=-4012, expire_ts=1727337648238611, current_time=1727337648308093) [2024-09-26 16:00:48.309132] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] acquire global snapshot fail(ret=-4012, gts_ahead=0, expire_ts=1727337648238611, now=[mts=1727337648309129], now0=[mts=1727337648309129], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:48.309136] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4012] acquire global snapshot fail(ret=-4012, tx={this:0x7f926915d960, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337648308093, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:48.309145] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=8][errcode=-4012] fail to get snapshot(ret=-4012, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915d960}) [2024-09-26 16:00:48.309149] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=4][errcode=-4012] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4012, session_id=1, *tx_desc={this:0x7f926915d960, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337648308093, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:48.309155] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=6] start stmt(ret=-4012, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926915d960, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337648308093, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337646308866, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915d960}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:48.309180] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=24][errcode=-4012] fail to start stmt(ret=-4012, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:48.309183] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4012] fail start stmt(ret=-4012) [2024-09-26 16:00:48.309185] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] execute plan failed(ret=-4012) [2024-09-26 16:00:48.309187] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=1][errcode=-4012] open result set failed(ret=-4012) [2024-09-26 16:00:48.309188] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=1][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:48.309190] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:48.309193] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:48.309196] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:48.309197] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=1][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:48.309199] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-09-26 16:00:48.309201] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7E-0-0] [lt=1][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:48.309204] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000337) [2024-09-26 16:00:48.309206] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:48.309209] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:48.309212] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:48.309213] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=1][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:48.309216] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:48.309221] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:48.309242] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:48.309244] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:48.309247] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:48.309249] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:48.309253] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:48.309255] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=1][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:48.309376] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7E-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:48.309383] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:48.309385] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:48.309402] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:48.309406] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:48.309680] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:48.309683] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:48.317915] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD5-0-0] [lt=2][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-09-26 16:00:48.317922] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD5-0-0] [lt=6][errcode=-4002] refresh sever list failed(ret=-4002) [2024-09-26 16:00:48.317924] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD5-0-0] [lt=2][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-09-26 16:00:48.320555] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:48.320562] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.325423] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:48.325433] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337648325420], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:48.325440] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337648325420], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:48.333579] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:48.333594] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337648333575) [2024-09-26 16:00:48.333628] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337648233064, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:48.333660] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223377569972, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:48.333667] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:48.333675] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:48.333680] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:48.333684] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:48.333690] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.333693] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.333695] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337648333632) [2024-09-26 16:00:48.351089] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:48.351100] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.363783] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337648325420]}) [2024-09-26 16:00:48.369628] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:48.381397] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29] success to notify_need_writing_throttling True [2024-09-26 16:00:48.381396] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D3D89-0-0] [lt=11][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:48.381417] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.383333] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1727337648383332, bytes: 14661448, bw: 0.003349 MB/s, add_ts: 1007986, add_bytes: 3540 [2024-09-26 16:00:48.388188] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=18] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:48.406759] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2648) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] [PALF_DUMP](palf_id=1, self="192.168.0.45:2882", [SlidingWindow]={palf_id:1, self:"192.168.0.45:2882", lsn_allocator:{max_log_id:183389608, max_lsn:629872191939, max_scn:{val:1727114270763524000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:629872191817}, reuse_lsn:{lsn:629872191939}, reserved_buffer_size:41943040, available_buffer_size:33554432, readable_begin_lsn:{lsn:629872191939}}, last_submit_lsn:{lsn:629872191817}, last_submit_end_lsn:{lsn:629872191939}, last_submit_log_id:183389608, last_submit_log_pid:19, max_flushed_lsn:{lsn:629872191817}, max_flushed_end_lsn:{lsn:629872191939}, max_flushed_log_pid:19, committed_end_lsn:{lsn:629872191939}, last_slide_log_id:183389608, last_slide_scn:{val:1727114270763524000, v:0}, last_slide_lsn:{lsn:629872191817}, last_slide_end_lsn:{lsn:629872191939}, last_slide_log_pid:19, last_slide_log_accum_checksum:2487403767, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f92b3ff0530}, [StateMgr]={this:0x7f92b3ff42b0, self:"192.168.0.45:2882", palf_id:1, role:"LEADER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:374}, leader:"192.168.0.45:2882", leader_epoch:195, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:629872191939}, scan_disk_log_finished:true, last_check_start_id:183389609, is_changing_config_with_arb:false, reconfirm_start_time_us:1727336098224413, palf_role_change_cb:0x7f92b3ffab70, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"192.168.0.45:2882", alive_paxos_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:374, prev:{version:1, config:{config_version:{proposal_id:373, config_seq:374}, log_sync_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:374, config_seq:375}, log_sync_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:19, prev_lsn:{lsn:629872191817}, prev_mode_pid:374}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:1, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, type:"STARTWORKING"}, state:0, checking_barrier:{prev_log_proposal_id:19, prev_lsn:{lsn:629872191817}, prev_end_lsn:{lsn:629872191939}, prev_mode_pid:374}, reconfig_barrier:{prev_log_proposal_id:19, prev_lsn:{lsn:629872191817}, prev_end_lsn:{lsn:629872191939}, prev_mode_pid:374}, persistent_config_version:{proposal_id:374, config_seq:375}, ms_ack_list:0, resend_config_version:{proposal_id:374, config_seq:375}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", register_time_us:1727336095809026, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1727336095809026, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f92b3ff1c70}, [ModeMgr]={palf_id:1, self:"192.168.0.45:2882", applied_mode_meta:{version:1, proposal_id:374, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:374, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:374, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:1726163155508348002, v:0}, min_block_id:9357, min_block_min_scn:{val:18446744073709551615, v:3}, base_lsn_for_block_gc:{lsn:627966418944}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:374}, log_config_meta:{version:3, proposal_id:374, prev:{version:1, config:{config_version:{proposal_id:373, config_seq:374}, log_sync_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:374, config_seq:375}, log_sync_memberlist:1{server:"192.168.0.45:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:19, prev_lsn:{lsn:629872191817}, prev_mode_pid:374}, log_snapshot_meta:{version:1, base_lsn:{lsn:627966418944}, prev_log_info:{log_id:-1, lsn:{lsn:18446744073709551615}, scn:{val:18446744073709551615, v:3}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:374, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:11440128}, readable_log_tail:{lsn:11440128}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:629912457216}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/observer/store/clog/tenant_1/1/meta", dir_fd:125, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:55664640, block_header_serialize_buf_:0x7f92b3ff6438, flashback_version:0}, log_storage:{log_tail:{lsn:629872191939}, readable_log_tail:{lsn:629872191939}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:9386, palf_id:1}, block_mgr:{log_dir:"/root/observer/store/clog/tenant_1/1/log", dir_fd:128, min_block_id:9357, max_block_id:9387, curr_writable_block_id:9386}, logical_block_size_:67104768, curr_block_writable_size_:40265277, block_header_serialize_buf_:0x7f92b3ff80b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:1727336098222784, this:0x7f92b3ff49f0}, [Reconfirm]={palf_id:1, self:"192.168.0.45:2882", state:"FINISHED", new_proposal_id:374, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:1, majority_max_log_server:"192.168.0.45:2882", majority_max_accept_pid:373, majority_max_lsn:{lsn:629872191939}, saved_end_lsn:{lsn:629872191939}, last_submit_prepare_req_time_us:1727336098224520, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f92b3ff4430}) [2024-09-26 16:00:48.406838] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2652) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=77] [PALF_DUMP](palf_id=1, self="192.168.0.45:2882", ack_info_list=[{member:{server:"192.168.0.45:2882", timestamp:1, flag:0}, last_ack_time_us:1727336098526269, last_flushed_end_lsn:{lsn:629872191939}}]) [2024-09-26 16:00:48.406844] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4290) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=5] current log_task status(palf_id=1, self="192.168.0.45:2882", log_id=183389609, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[{member:{server:"192.168.0.45:2882", timestamp:1, flag:0}, last_ack_time_us:1727336098526269, last_flushed_end_lsn:{lsn:629872191939}}]) [2024-09-26 16:00:48.406855] WDIAG [STORAGE] get_server_region (ob_locality_manager.cpp:399) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4018] fail to get server region(ret=-4018, server="192.168.0.45:2882") [2024-09-26 16:00:48.406862] WDIAG [CLOG] get_server_region (ob_locality_adapter.cpp:61) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4018] get_server_region failed(server="192.168.0.45:2882") [2024-09-26 16:00:48.406865] WDIAG [PALF] update_self_region_ (palf_handle_impl.cpp:408) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4018] get_server_region failed(ret=-4018, this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}) [2024-09-26 16:00:48.406867] WDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2658) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4018] update_region failed(ret=-4018, this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}) [2024-09-26 16:00:48.411638] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:48.411645] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.434141] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:48.434151] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337648434138) [2024-09-26 16:00:48.434166] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337648333632, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:48.434183] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223377670647, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:48.434189] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:48.434196] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:48.434201] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:48.434203] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:48.434208] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.434210] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.434212] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337648434170) [2024-09-26 16:00:48.441933] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:48.441940] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.452747] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:33) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=2] === Run print tenant memory usage task === [2024-09-26 16:00:48.452798] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 411,398,144 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 180,654,080 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 88,080,384 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 12,582,912 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 17,969,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 11,292,672 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 90,333,184 limit= 9,223,372,036,854,775,807 [2024-09-26 16:00:48.452891] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:176) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 180,654,080 used= 173,119,648 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] hold= 33,574,912 used= 33,554,464 count= 1 avg_used= 33,554,464 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 18,894,848 used= 18,874,392 count= 1 avg_used= 18,874,392 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 12,460,032 used= 11,125,608 count= 169 avg_used= 65,832 block_cnt= 169 chunk_cnt= 29 mod=[T]ObSessionDIB [MEMORY] hold= 10,776,576 used= 10,735,904 count= 11 avg_used= 975,991 block_cnt= 11 chunk_cnt= 7 mod=IoControl [MEMORY] hold= 8,585,216 used= 8,566,048 count= 1 avg_used= 8,566,048 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 5,251,072 used= 5,238,784 count= 32 avg_used= 163,712 block_cnt= 32 chunk_cnt= 9 mod=Iteratorget_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6499, mem_used=12478464) [2024-09-26 16:00:48.564035] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=6] gc stale ls task succ [2024-09-26 16:00:48.564051] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:48.564071] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.565006] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337648325420]}) [2024-09-26 16:00:48.569868] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:48.594310] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:48.594317] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.601153] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.601190] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.601200] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.602111] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.609080] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:783) [2618][AutoLSLocRpc][T0][YB42C0A8002D-0006230057E8588A-0-0] [lt=14] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.0.45:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, location:{server:"192.168.0.45:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:374}}]) [2024-09-26 16:00:48.624394] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:48.624404] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.634910] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:48.634925] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337648634905) [2024-09-26 16:00:48.634932] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337648534777, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:48.634952] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223377870792, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:48.634958] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:48.634967] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:48.634972] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:48.634974] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:48.634980] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.634982] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.634984] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337648634935) [2024-09-26 16:00:48.636430] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:48.636441] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:48.636446] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:48.636470] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=23][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:48.636474] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:48.636476] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:48.636479] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:48.636482] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:48.636484] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:48.636486] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:48.636488] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:48.636490] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:48.636493] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:48.636495] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:48.636501] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:48.636504] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:48.636507] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:48.636509] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:48.636511] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, ret=-5019) [2024-09-26 16:00:48.636514] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:48.636516] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:48.636522] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:48.636528] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:48.636530] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:48.636532] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:48.636538] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:48.636541] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:48.636544] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD5-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-09-26 16:00:48.636546] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:48.636548] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:48.636551] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:48.636553] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337648636360, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:48.636557] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:48.636559] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f92b4056738, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, columns_str="zone") [2024-09-26 16:00:48.636564] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:48.636594] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:634) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f92b4056738, where_condition="where svr_ip='192.168.0.45' and svr_port=2882", zone_name_holder=) [2024-09-26 16:00:48.636598] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:567) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:48.636600] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:576) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:48.636602] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:48.636608] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:48.654831] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=30] success to notify_need_writing_throttling True [2024-09-26 16:00:48.654841] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.661670] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:48.665612] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337648325420]}) [2024-09-26 16:00:48.665621] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855E9-0-0] [lt=3] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:48.665644] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855EB-0-0] [lt=8] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:48.670640] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:48.670752] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:48.671376] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=817882, clean_num=31457) [2024-09-26 16:00:48.685151] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:48.685169] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.708563] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:48.715440] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:48.715448] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.718351] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:445) [2901][T1_SSTableDefra][T1][Y0-0000000000000000-0-0] [lt=2] skip defragment task(ret=0, need_defragment=false) [2024-09-26 16:00:48.718372] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:485) [2901][T1_SSTableDefra][T1][Y0-0000000000000000-0-0] [lt=20] successfully defragment data blocks(ret=0, rewrite_cnt=0, block_used_size_.count()=42) [2024-09-26 16:00:48.718385] INFO [STORAGE.BLKMGR] defragment (ob_shared_macro_block_manager.cpp:503) [2901][T1_SSTableDefra][T1][Y0-0000000000000000-0-0] [lt=4] successfully defragment data blocks(rewrite_cnt=0) [2024-09-26 16:00:48.734942] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:48.734968] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337648734939) [2024-09-26 16:00:48.734974] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337648634935, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:48.734992] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223377971432, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:48.734998] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:48.735005] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:48.735019] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:48.735023] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:48.735029] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.735031] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.735033] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337648734977) [2024-09-26 16:00:48.735897] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=6] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:48.745715] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:48.745723] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.761337] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1073) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=4] replay status is not follower(min_unreplayed_lsn={lsn:629872191939}, base_lsn={lsn:627966418944}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:00:48.761381] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=44] get_replay_process success(id={id:1}, replayed_log_size=1905772995, unreplayed_log_size=0) [2024-09-26 16:00:48.761395] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:159) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=4] dump tenant replay process(tenant_id=1, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=1817, last_replayed_log_size(MB)=1817, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-09-26 16:00:48.766190] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337648325420]}) [2024-09-26 16:00:48.770391] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=18] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=786400, replace_num=15728) [2024-09-26 16:00:48.770408] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=16] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:48.775972] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:48.775988] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.779847] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337648325420]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:48.779862] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=14][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:48.806256] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:48.806262] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.820942] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.820999] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.821850] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=7] PNIO [ratelimit] time: 1727337648821849, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000026, add_bytes: 0 [2024-09-26 16:00:48.821890] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.821900] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.821906] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.821913] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.821919] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.821924] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.821928] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.821933] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.823822] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:48.825871] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:48.825879] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337648825869], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:48.825885] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337648825869], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:48.825889] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:48.825892] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:48.826736] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [2910][T1_RFLWorker][T1][YB42C0A8002D-0006230055684B24-0-0] [lt=8] ObRemoteFetchWorker is running(thread_index=0) [2024-09-26 16:00:48.835550] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:48.835559] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337648835547) [2024-09-26 16:00:48.835564] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337648734977, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:48.835580] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223378072017, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:48.835585] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:48.835592] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:48.835597] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:48.835600] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:48.835605] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.835608] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.835610] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337648835567) [2024-09-26 16:00:48.836531] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:48.836538] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.866792] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:48.866791] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337648825869]}) [2024-09-26 16:00:48.866799] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.871790] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:48.871912] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash time detail, (compute_wash_size_time=54, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:48.872500] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=849339, clean_num=31457) [2024-09-26 16:00:48.897077] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:48.897084] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.921506] INFO [CLOG] run1 (ob_garbage_collector.cpp:1338) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=2] Garbage Collector is running(seq_=345, gc_interval=10000000) [2024-09-26 16:00:48.921524] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1473) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] self is leader, skip it(ls->get_ls_id()={id:1}) [2024-09-26 16:00:48.921533] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1431) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=4] gc_check_member_list_ cost time(ret=0, time_us=17) [2024-09-26 16:00:48.921540] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=5] execute_gc cost time(ret=0, time_us=0) [2024-09-26 16:00:48.921543] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=2] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-09-26 16:00:48.921548] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2024-09-26 16:00:48.921551] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=2] execute_gc cost time(ret=0, time_us=4) [2024-09-26 16:00:48.921553] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=1] ObSafeDestroyHandler start process [2024-09-26 16:00:48.927371] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:48.927380] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.936178] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:48.936191] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337648936166) [2024-09-26 16:00:48.936197] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337648835567, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:48.936224] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=223378172625, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:48.936230] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:48.936238] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:48.936243] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:48.936246] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:48.936253] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.936256] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:48.936258] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337648936210) [2024-09-26 16:00:48.958638] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:48.958647] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:48.959884] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-09-26 16:00:48.959895] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-09-26 16:00:48.959899] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-09-26 16:00:48.959904] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=5][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-09-26 16:00:48.959907] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:48.959910] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:48.959913] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-09-26 16:00:48.959915] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:48.959918] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:48.959919] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:48.959921] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:48.959923] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:48.959926] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:48.959928] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:48.959935] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:48.959938] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:48.959941] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:48.959943] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:48.959945] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:48.959948] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:48.959950] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:48.959956] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:48.959961] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:48.959964] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:48.959966] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:48.959972] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:48.959975] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:48.959978] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2024-09-26 16:00:48.959981] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:48.959983] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:48.959985] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:48.959987] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337648959811, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:48.959990] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:48.959993] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1111) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:48.960018] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:801) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:48.960021] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:67) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=3][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:48.960024] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:436) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:48.960026] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:95) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:48.960029] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:109) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:48.960239] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-09-26 16:00:48.960243] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=3][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2024-09-26 16:00:48.960245] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2024-09-26 16:00:48.960249] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2024-09-26 16:00:48.960251] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:48.960252] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:48.960255] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_tenant' doesn't exist [2024-09-26 16:00:48.960256] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:48.960258] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:48.960259] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:48.960260] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:48.960262] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:48.960263] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:48.960265] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:48.960267] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:48.960268] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:48.960270] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:48.960271] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:48.960273] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:48.960274] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:48.960276] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:48.960278] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:48.960283] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:48.960284] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:48.960285] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:48.960288] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:48.960290] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:48.960293] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, aret=-5019, ret=-5019) [2024-09-26 16:00:48.960294] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:00:48.960296] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:48.960297] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:48.960299] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337648960218, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:00:48.960301] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:48.960303] WDIAG [SHARE] read_tenants (ob_unit_table_operator.cpp:1174) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] execute sql failed(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:48.960314] WDIAG [SHARE] get_tenants (ob_unit_table_operator.cpp:108) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=1][errcode=-5019] fail read tenants(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:48.960316] WDIAG [SHARE] get_tenants (ob_unit_getter.cpp:198) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] ut_operator get_resource_pools failed(ret=-5019) [2024-09-26 16:00:48.960318] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:125) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2][errcode=-5019] get cluster tenants fail(ret=-5019) [2024-09-26 16:00:48.960331] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:325) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2] cal_all_part_disk_default_percentage succ(data_dir="/root/observer/store/sstable", clog_dir="/root/observer/store/clog", shared_mode=true, data_disk_total_size=99765694464, data_disk_default_percentage=60, clog_disk_total_size=99765694464, clog_disk_default_percentage=30) [2024-09-26 16:00:48.960335] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:361) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=3] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=99765694464, disk_size=15032385536) [2024-09-26 16:00:48.960337] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=2] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=99765694464) [2024-09-26 16:00:48.960340] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=3] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=99765694464, this={dir::"/root/observer/store/clog/log_pool", dir_fd:15, meta_fd:16, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:9388, max_block_id:9581, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2024-09-26 16:00:48.960347] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:133) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD6-0-0] [lt=6] refresh tenant config(tenants=[], ret=-5019) [2024-09-26 16:00:48.967396] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337648825869]}) [2024-09-26 16:00:48.971321] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=5] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:48.981072] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:48.988961] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:48.988967] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.019217] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:49.019223] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.022263] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500781, support_ssl=false) [2024-09-26 16:00:49.022288] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221500781, sess.client_addr_="127.0.0.1:36358") [2024-09-26 16:00:49.022296] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500781, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:49.022314] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:49.022325] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2688][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:49.022393] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA2-0-0] [lt=8] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:49.022588] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA2-0-0] [lt=9][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:49.022597] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA2-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"{D"{gp!MA?Vtf"tD7$OB"}, ret=-4043) [2024-09-26 16:00:49.022603] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA2-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:49.022608] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA2-0-0] [lt=4][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:49.022616] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA2-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:49.022637] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA2-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500781, proxy_sessid:0}) [2024-09-26 16:00:49.022655] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA2-0-0] [lt=4] mark session id unused(sessid=3221500781) [2024-09-26 16:00:49.022660] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA2-0-0] [lt=3][errcode=0] server close connection(sessid=3221500781, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:49.022664] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA2-0-0] [lt=4][errcode=-4018] get session fail(ret=-4018, sessid=3221500781, proxy_sessid=0) [2024-09-26 16:00:49.022666] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA2-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:49.022669] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA2-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500781, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:49.022696] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500781, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337649022115, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:49.022714] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221500781, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:49.022717] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500781, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337649022115, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:49.022721] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500781, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:49.030004] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591A-0-0] [lt=32][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:49.030015] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591A-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:49.030023] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591A-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:49.030028] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591A-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:49.036732] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:49.036739] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:49.036744] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337649036730) [2024-09-26 16:00:49.036749] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337648936210, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:49.036754] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:49.036763] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:49.036766] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:49.037058] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885760-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:49.037068] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885760-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:49.037073] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885760-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:49.037076] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885760-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:49.037098] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885760-0-0] [lt=20][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:49.037101] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885760-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:49.037104] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885760-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:49.037106] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885760-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:49.037108] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885760-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:49.037110] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885760-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:49.037112] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885760-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:49.037142] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885760-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:49.037172] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337649037172, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=412, wlock_time=8, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:49.037182] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:49.037197] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223378273206, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:49.037202] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:49.037209] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:49.037214] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:49.037216] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:49.037222] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.037224] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.037226] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337649037187) [2024-09-26 16:00:49.049314] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:49.049323] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.049333] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:49.049647] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588A-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:49.049659] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588A-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:49.049695] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588A-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:49.061621] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:49.061906] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6E-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:49.061930] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6E-0-0] [lt=22][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:49.061955] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E6E-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:49.062486] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337648062043]) [2024-09-26 16:00:49.062496] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337648062043], now0=[mts=1727337648062043], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:49.062502] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337648061976, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:49.062519] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:49.062529] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337648061976, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:49.062537] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337648061976, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:49.062553] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:49.062558] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:49.062561] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:49.062563] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:49.062566] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:49.062569] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=11, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:49.062573] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=11000, remain_us=17931759, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=11, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:49.068009] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=33] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337648825869]}) [2024-09-26 16:00:49.072993] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=16] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:49.073113] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=71, refresh_score_time=28, wash_time=2) [2024-09-26 16:00:49.073595] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:11, local_retry_times:11, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:49.073610] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:49.073615] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:49.073618] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:49.073626] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:49.073666] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:49.073670] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:49.073759] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=880796, clean_num=31457) [2024-09-26 16:00:49.079760] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:49.079768] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.085545] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:49.088530] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10696.00) [2024-09-26 16:00:49.088539] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:49.088550] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:49.088555] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:49.110014] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:49.110022] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.137362] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:49.137376] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337649137359) [2024-09-26 16:00:49.137382] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337649036752, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:49.137400] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223378373785, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:49.137416] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:49.137425] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:49.137430] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:49.137433] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:49.137439] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.137442] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.137444] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337649137385) [2024-09-26 16:00:49.140286] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:49.140303] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.153946] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-09-26 16:00:49.154282] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD5-0-0] [lt=8] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-09-26 16:00:49.154295] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD5-0-0] [lt=12][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:49.154324] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD5-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:49.168645] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337648825869]}) [2024-09-26 16:00:49.168662] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D906-0-0] [lt=12][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337649168627}) [2024-09-26 16:00:49.170576] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:49.170589] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.171497] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:49.184230] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=26] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:49.200827] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:49.200834] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.205954] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:49.205968] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:49.205973] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:49.231010] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:49.231016] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.237455] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:49.237471] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337649237452) [2024-09-26 16:00:49.237477] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337649137385, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:49.237494] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223378473516, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:49.237500] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:49.237507] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:49.237512] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:49.237515] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:49.237521] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.237523] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.237525] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337649237481) [2024-09-26 16:00:49.246646] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:49.246657] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:49.246670] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=12][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:49.246673] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:49.246681] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=7][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:49.246684] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:49.246687] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:49.246689] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:49.246692] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:49.246694] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:49.246696] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:49.246699] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:49.246719] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-0006230056485899-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:49.261384] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:49.261390] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.268750] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337648825869]}) [2024-09-26 16:00:49.274603] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:49.274710] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=63, refresh_score_time=17, wash_time=2) [2024-09-26 16:00:49.275315] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=912253, clean_num=31457) [2024-09-26 16:00:49.282596] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:161, tid:2919}]) [2024-09-26 16:00:49.291489] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:49.291497] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.309230] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337650239409, now=[mts=1727337648309678]) [2024-09-26 16:00:49.309240] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337650239409, now=[mts=1727337648309678], now0=[mts=1727337648309678], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:49.309247] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926915f700, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337648309168, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:49.309264] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f9263676038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915f700}) [2024-09-26 16:00:49.309272] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926915f700, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337648309168, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:49.309280] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926915f700, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337648309168, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337648309660, use_das=false, nested_level=0, session={this:0x7f9263676038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915f700}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:49.309296] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:49.309301] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:49.309304] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:49.309307] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:49.309309] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:49.309312] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:49.309317] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=1000093, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337650309409) [2024-09-26 16:00:49.309320] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:49.309328] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:49.309330] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=1][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:49.309331] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:49.309337] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:49.309370] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:49.309373] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:49.321586] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:49.321594] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.326210] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.326218] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337649326208], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:49.326225] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337649326208], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:49.337479] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223378572943, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:49.337489] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:49.337505] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:49.337513] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:49.337520] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:49.337526] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.337530] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.337532] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337649337463) [2024-09-26 16:00:49.351716] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:49.351727] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.368805] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337649326208]}) [2024-09-26 16:00:49.369934] INFO [STORAGE] runTimerTask (ob_tenant_tablet_scheduler.cpp:192) [2947][T1_InfoPoolResi][T1][Y0-0000000000000000-0-0] [lt=8] InfoPoolResizeTask(cost_ts=4) [2024-09-26 16:00:49.371575] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:49.382126] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:49.382158] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.387875] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D3DE8-0-0] [lt=18][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:49.388219] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [2645][Blacklist][T0][Y0-0000000000000000-0-0] [lt=7] blacklist_loop exec finished(cost_time=10, is_enabled=true, send_cnt=0) [2024-09-26 16:00:49.388296] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:49.388308] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=4] current monitor number(seq_=-1) [2024-09-26 16:00:49.390391] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [2916][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14] start to check and clear obj lock(ls->get_ls_id()={id:1}) [2024-09-26 16:00:49.390449] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337649390448, bytes: 14665270, bw: 0.003619 MB/s, add_ts: 1007116, add_bytes: 3822 [2024-09-26 16:00:49.403417] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:49.403432] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:49.403438] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=5][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:49.403442] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:49.403447] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=5][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:49.403451] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=3] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:49.403454] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:49.403456] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:49.403460] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:49.403462] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:49.403466] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:49.403468] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:49.403470] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:49.403473] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:49.403478] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:49.403480] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:49.403488] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:49.403490] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:49.403494] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:49.403496] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:49.403498] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-09-26 16:00:49.403522] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:49.403529] WDIAG [SERVER] do_process (obmp_query.cpp:771) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=6][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-09-26 16:00:49.403535] WDIAG [SERVER] do_process (obmp_query.cpp:903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, session={this:0x7f92a230bdf8, id:3221487638, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-09-26 16:00:49.403546] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F50-0-0] [lt=7] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:49.412243] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:49.412250] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.434728] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] ====== tenant freeze timer task ====== [2024-09-26 16:00:49.434741] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:49.434772] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337645627740, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337645627695, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337645627760, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:49.434794] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:49.434855] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DD-0-0] [lt=4] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:49.434866] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DD-0-0] [lt=8] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:49.434871] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DD-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:49.434875] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DD-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:49.434878] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DD-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:49.434881] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DD-0-0] [lt=3][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:49.434884] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DD-0-0] [lt=3][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:49.434893] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DD-0-0] [lt=9] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:49.434895] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DD-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:49.434898] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DD-0-0] [lt=3] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:49.434901] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DD-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:49.434937] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851DD-0-0] [lt=6] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:49.437493] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:49.437504] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337649437491) [2024-09-26 16:00:49.437508] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337649237481, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:49.437525] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223378673387, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:49.437531] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:49.437538] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:49.437542] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:49.437544] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:49.437549] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.437551] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.437553] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337649437512) [2024-09-26 16:00:49.442322] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:49.442330] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.459680] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.459686] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.459690] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.460446] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.460454] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.460460] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.460498] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.460503] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.460506] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.461412] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.461412] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.461417] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.461419] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.461416] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.461445] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=29][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.461979] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.461985] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.461987] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.462023] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.462026] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.462028] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.462776] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.462781] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.462783] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.462778] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.462789] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.462792] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.463391] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.463392] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.463394] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.463395] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.463396] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.463397] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.463922] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.463925] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.463927] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.463953] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.463957] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.463959] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.464654] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.464659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.464661] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.464666] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.464669] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.464671] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.465508] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.465514] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.465518] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:49.465522] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.466043] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.466270] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.467117] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.467132] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.467704] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.467840] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.468452] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.468480] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.468837] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [2940][T1_ArcFetcher][T1][YB42C0A8002D-0006230055E84B24-0-0] [lt=7] ObArchiveFetcher is running(thread_index=0) [2024-09-26 16:00:49.469440] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.469442] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.469462] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337649326208]}) [2024-09-26 16:00:49.470380] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.470377] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.471367] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=26][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.471373] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.472340] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.472377] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.472415] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:49.472423] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.472970] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.472973] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.473529] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.473604] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.474282] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.474286] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.474814] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.474861] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.475336] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.475394] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.475588] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:49.475690] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=61, refresh_score_time=17, wash_time=1) [2024-09-26 16:00:49.476284] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=943710, clean_num=31457) [2024-09-26 16:00:49.476315] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.476350] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.477217] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.477240] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.477750] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.477750] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.478565] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.478604] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.480150] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.480151] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.482123] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.482152] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.484085] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.484128] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.484683] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.484865] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.486025] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.486062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.487004] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.487013] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.488948] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.489017] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.489925] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.489928] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.490897] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.490965] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.491874] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.491881] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.492882] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.492901] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.493821] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.493824] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.494734] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.494829] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.495781] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.495794] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.496761] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.496758] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.497731] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.498725] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.498746] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.499344] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.499345] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.500671] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.500655] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.501643] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.501657] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.502613] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.503616] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.503628] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:49.503634] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.503633] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.504581] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.504604] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.505564] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.505583] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.506533] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.506544] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.507230] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:49.527173] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591B-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:49.527183] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591B-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:49.527192] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591B-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:49.527198] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591B-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:49.533864] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:49.533871] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.537519] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:49.537529] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337649537516) [2024-09-26 16:00:49.537534] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337649437512, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:49.537539] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:49.537549] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:49.537552] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:49.537905] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:49.537918] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:49.537922] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:49.537926] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:49.537929] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:49.537931] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:49.537935] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:49.537937] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:49.537939] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:49.537941] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:49.537943] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:49.537945] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:49.537948] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:49.537950] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:49.537956] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:49.537958] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:49.537961] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:49.537963] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:49.537965] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:49.537968] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:49.537971] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:49.537976] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:49.537981] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:49.537983] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:49.537985] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:49.537990] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:49.537994] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:49.537996] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885761-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:49.537998] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:49.538001] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:49.538003] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:49.538005] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337649537823, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:49.538009] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:49.538011] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:49.538041] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:49.538043] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337649538042, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=497, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:49.538050] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:49.538052] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:49.538068] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223378773938, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:49.538072] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:49.538078] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:49.538082] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:49.538084] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:49.538089] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.538091] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.538092] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337649538056) [2024-09-26 16:00:49.544764] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6511, mem_used=12478464) [2024-09-26 16:00:49.564161] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:49.564168] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=19] gc stale ls task succ [2024-09-26 16:00:49.564169] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.570073] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337649326208]}) [2024-09-26 16:00:49.571689] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:49.594419] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:49.594426] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.602282] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.602283] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.602296] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.602296] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.624738] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:49.624744] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.637845] WDIAG [PALF] convert_to_ts (scn.cpp:265) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2024-09-26 16:00:49.637855] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1727337649637844, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1727337649637844, force_self_check=false, weak_read_refresh_interval=100000) [2024-09-26 16:00:49.637870] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:49.637875] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337649637868) [2024-09-26 16:00:49.637879] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337649537537, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:49.637895] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223378873947, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:49.637900] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:49.637908] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:49.637912] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:49.637915] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:49.637921] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.637924] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.637926] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337649637882) [2024-09-26 16:00:49.638059] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:49.654988] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:49.654997] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.661835] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:49.670634] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337649326208]}) [2024-09-26 16:00:49.676630] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:49.676734] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:49.677348] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=975167, clean_num=31457) [2024-09-26 16:00:49.685274] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:49.685282] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.713401] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:49.715515] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:49.715520] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.738005] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:49.738019] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337649738003) [2024-09-26 16:00:49.738024] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337649637882, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:49.738055] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get_max_applied_scn(scn={val:1727114270763524000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:374, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:1727114270763524000, v:0}, max_applied_cb_scn_:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:49.738064] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get_min_unreplayed_log_info(lsn={lsn:629872191939}, scn={val:1727114270763524001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:00:49.738095] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=223378974119, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:49.738101] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:49.738107] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:49.738111] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:49.738114] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:49.738122] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.738125] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.738127] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337649738045) [2024-09-26 16:00:49.745802] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:49.745809] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.759081] INFO [STORAGE.TRANS] print_statistics (ob_xa_define.cpp:467) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=19] xa statistics(total_active_xa_ctx_count=0, total_standby_clearup_count=0, total_success_xa_start=0, total_failure_xa_start=0, total_success_xa_prepare=0, total_failure_xa_prepare=0, total_success_xa_1pc_commit=0, total_failure_xa_1pc_commit=0, total_success_xa_2pc_commit=0, total_failure_xa_2pc_commit=0, total_failure_xa_rollback=0, total_success_dblink_promotion=0, total_failure_dblink_promotion=0, total_success_dblink=0, total_failure_dblink=0) [2024-09-26 16:00:49.771218] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337649326208]}) [2024-09-26 16:00:49.772186] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:49.776114] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:49.776121] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.781961] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=4] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-09-26 16:00:49.781972] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=8] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-09-26 16:00:49.781977] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=4] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=690) [2024-09-26 16:00:49.806367] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:49.806387] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.821099] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.821157] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.821184] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.821192] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.821196] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.821200] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.821208] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.821215] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.821221] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.821769] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.821785] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:49.826998] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:49.827009] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337649826995], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:49.827019] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337649826995], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:49.827024] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:49.827029] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:49.828864] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337649828864, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007015, add_bytes: 0 [2024-09-26 16:00:49.836673] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:49.836681] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.838612] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:49.838632] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337649838600) [2024-09-26 16:00:49.838638] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337649738045, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:49.838656] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223379074123, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:49.838664] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:49.838683] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:49.838688] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:49.838692] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:49.838698] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.838701] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.838702] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337649838641) [2024-09-26 16:00:49.867004] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:49.867013] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.868291] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, table_name.ptr()="data_size:16, data:5F5F616C6C5F6D657267655F696E666F", ret=-5019) [2024-09-26 16:00:49.868306] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, ret=-5019) [2024-09-26 16:00:49.868312] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_merge_info, db_name=oceanbase) [2024-09-26 16:00:49.868317] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_merge_info) [2024-09-26 16:00:49.868322] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:49.868325] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:49.868330] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_merge_info' doesn't exist [2024-09-26 16:00:49.868333] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:49.868336] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:49.868339] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=3][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:49.868342] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:49.868345] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:49.868351] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=5][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:49.868354] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:49.868362] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:49.868365] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:49.868369] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:49.868372] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:49.868375] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_merge_info WHERE tenant_id = '1', ret=-5019) [2024-09-26 16:00:49.868397] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:49.868402] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:49.868410] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=6][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:49.868418] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:49.868421] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:49.868423] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:49.868432] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:49.868436] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:49.868440] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885086-0-0] [lt=4][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, aret=-5019, ret=-5019) [2024-09-26 16:00:49.868444] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:49.868447] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:49.868451] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:49.868454] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337649868208, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:49.868458] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:49.868462] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:49) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:49.868498] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:890) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] failed to load global merge info(ret=-5019, ret="OB_TABLE_NOT_EXIST", global_merge_info={tenant_id:1, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2024-09-26 16:00:49.868514] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1005) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] fail to refresh merge info(tmp_ret=-5019, tmp_ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:49.868866] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:49.868872] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:49.871459] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337649826995]}) [2024-09-26 16:00:49.877565] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=16] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2250211328, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:49.877713] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=52, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:49.878558] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1006624, clean_num=31457) [2024-09-26 16:00:49.898203] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:49.898224] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.918692] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] ====== checkpoint timer task ====== [2024-09-26 16:00:49.919094] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] succ to get rec scn(*this={this:0x7f92a6a04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1726077600526835000, v:0}, last_push_gc_task_ts_:1727334198745636, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:49.919119] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=24] tx ctx memtable get rec scn(this={ObITable:{this:0x7f92a22a4080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f92a22a4080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:49.919131] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1723174350926847000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-09-26 16:00:49.919148] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:49.919170] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=20] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}, min_rec_scn={val:1726163586617842000, v:0}, ls_id_={id:1}) [2024-09-26 16:00:49.919183] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:49.919186] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:49.919189] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:49.919195] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=1] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1726163586617842000, v:0}, checkpoint_scn_in_ls_meta={val:1726163586617842000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-09-26 16:00:49.919211] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}, curr_base_lsn={lsn:627966418944}, new_base_lsn={lsn:627966418944}, lsn={lsn:627966418944}) [2024-09-26 16:00:49.919217] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:627966418944}, ls->get_ls_id()={id:1}) [2024-09-26 16:00:49.919220] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-09-26 16:00:49.928376] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:49.928383] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.940261] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:49.940273] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337649940259) [2024-09-26 16:00:49.940279] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337649838641, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:49.940293] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223379173940, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:49.940299] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:49.940306] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:49.940311] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:49.940314] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:49.940320] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.940323] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:49.940325] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337649940282) [2024-09-26 16:00:49.958743] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:49.958750] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:49.971477] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337649826995]}) [2024-09-26 16:00:49.972407] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:49.977759] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [2708][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=206] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"2GB", log_disk_size:"2GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, mode:0, create_timestamp:1710828706488537, is_removed:false}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=15, log_id=1828900, offset=14449774}, ls_meta_entry:[10007](ver=0,mode=0,seq=609892), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:2}, create_status:1}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:49, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:80050099, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:1730, recv_lp_rpc_cnt:0, recv_mysql_cnt:1148, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:9736902, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:group_id = 2,queue_size = 0,recv_req_cnt = 7544,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334211521784 group_id = 10,queue_size = 0,recv_req_cnt = 3432,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334216415596 group_id = 9,queue_size = 0,recv_req_cnt = 7056,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334199387293 group_id = 1,queue_size = 0,recv_req_cnt = 4466,min_worker_cnt = 8,max_worker_cnt = 8,worker_cnt = 0,token_change = 1727336105580026 , rpc_stat_info: pcode=0x150a:cnt=20 pcode=0x150b:cnt=20 pcode=0x4a9:cnt=10 pcode=0x113:cnt=5, token_change_ts:1727334199363389, tenant_role:1}) [2024-09-26 16:00:49.977962] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [2708][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=202] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1727334198122806, is_removed:false}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:2}, create_status:1}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:12, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:2118786, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3245372, workers:12, nesting workers:0, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1727334198125585, tenant_role:0}) [2024-09-26 16:00:49.981252] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:49.984142] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=32] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-09-26 16:00:49.984164] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=19] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:49.984169] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=4] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:49.984173] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=3] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:49.985855] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] Dumping All Cache Objs(alloc_obj_list.count()=3, alloc_obj_list=[{obj_id:78, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157779}, {obj_id:79, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:5, added_to_lc:true, mem_used:175172}]) [2024-09-26 16:00:49.985867] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=11] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:49.989049] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:49.989055] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.019304] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:50.019313] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.024386] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221500788, support_ssl=false) [2024-09-26 16:00:50.024421] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221500788, sess.client_addr_="127.0.0.1:36360") [2024-09-26 16:00:50.024429] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] accept one succ(*s={this:0x7f9264604030, session_id:3221500788, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:50.024478] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:50.024489] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=5] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:50.024526] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA1-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:50.024595] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:50.024616] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=20][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-09-26 16:00:50.024622] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:50.024626] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-09-26 16:00:50.024630] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:50.024633] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:50.024637] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-09-26 16:00:50.024640] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:50.024642] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:50.024644] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:50.024647] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=3][errcode=-5019] resolve table failed(ret=-5019) [2024-09-26 16:00:50.024649] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:2822) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-09-26 16:00:50.024651] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2676) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] resolve joined table failed(ret=-5019) [2024-09-26 16:00:50.024653] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:50.024656] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:50.024658] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:50.024660] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:50.024667] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:50.024670] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:50.024675] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:50.024679] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:50.024681] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-09-26 16:00:50.024684] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:50.024687] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:50.024693] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:50.024698] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:50.024701] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:50.024698] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA1-0-0] [lt=3][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:50.024703] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:50.024704] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA1-0-0] [lt=5][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"EcCcL/GXdmv+mf7!=P[X"}, ret=-4043) [2024-09-26 16:00:50.024709] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:50.024710] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA1-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:50.024713] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA2-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:50.024713] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA1-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:50.024716] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [3024][BlackListServic][T0][YB42C0A8002D-0006230056284FA2-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-09-26 16:00:50.024719] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:50.024722] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:50.024721] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA1-0-0] [lt=3] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:50.024726] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:50.024730] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337650024492, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:50.024733] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:50.024735] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:222) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:50.024739] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:398) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=4] start to print blacklist info [2024-09-26 16:00:50.024742] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA1-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500788, proxy_sessid:0}) [2024-09-26 16:00:50.024747] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA1-0-0] [lt=4] mark session id unused(sessid=3221500788) [2024-09-26 16:00:50.024750] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA1-0-0] [lt=1][errcode=0] server close connection(sessid=3221500788, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:50.024753] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA1-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500788, proxy_sessid=0) [2024-09-26 16:00:50.024755] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA1-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:50.024758] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA1-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500788, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:50.024764] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=4] ls blacklist refresh finish(cost_time=568) [2024-09-26 16:00:50.024790] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f9264604030, session_id:3221500788, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337650024283, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:50.024811] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221500788, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:50.024815] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9264604030, session_id:3221500788, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337650024283, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:50.024829] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221500788, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:50.030222] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591C-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:50.030233] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591C-0-0] [lt=11][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:50.030243] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591C-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:50.030248] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591C-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:50.040846] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:50.040855] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:50.040862] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337650040843) [2024-09-26 16:00:50.040891] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337649940282, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:50.040900] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:50.040909] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:50.040912] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:50.041203] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:50.041213] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:50.041217] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:50.041220] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:50.041224] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:50.041226] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:50.041230] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:50.041233] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:50.041235] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:50.041237] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:50.041238] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:50.041240] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:50.041242] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:50.041244] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:50.041249] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:50.041251] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:50.041253] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:50.041255] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:50.041257] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:50.041259] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:50.041262] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:50.041267] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:50.041272] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:50.041274] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:50.041275] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:50.041280] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:50.041283] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:50.041289] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885762-0-0] [lt=6][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:50.041292] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:50.041295] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:50.041297] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:50.041299] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337650041136, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:50.041302] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:50.041305] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:50.041327] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:50.041332] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337650041331, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=426, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:50.041338] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:50.041341] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:50.041356] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223379276379, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.041360] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:50.041368] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:50.041373] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:50.041375] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:50.041381] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.041383] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.041385] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337650041344) [2024-09-26 16:00:50.049576] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:50.049584] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.050748] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:50.050759] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:50.050763] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:50.050766] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:50.050769] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:50.050771] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:50.050774] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:50.050776] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:50.050778] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:50.050783] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=5][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:50.050785] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:50.050787] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:50.050789] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:50.050801] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=12][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:50.050807] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:50.050809] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.050821] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588B-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:50.062276] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=11] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:50.062535] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E70-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:50.062541] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E70-0-0] [lt=5][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.062557] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E70-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:50.071495] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337649826995]}) [2024-09-26 16:00:50.073406] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337649073664]) [2024-09-26 16:00:50.073417] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337649073664], now0=[mts=1727337649073664], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:50.073434] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337649073007, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:50.073452] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:50.073460] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337649073007, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:50.073469] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337649073007, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:50.073486] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:50.073491] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:50.073512] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=20][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:50.073515] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:50.073517] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:50.073521] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=12, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:50.073525] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=12000, remain_us=16920807, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:50.078753] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:50.078866] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash time detail, (compute_wash_size_time=56, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:50.079497] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1038081, clean_num=31457) [2024-09-26 16:00:50.079671] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:50.079679] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.079690] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:50.085626] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:12, local_retry_times:12, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:50.085642] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:50.085645] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:50.085646] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:50.085659] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:50.085700] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:50.085704] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:50.086602] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=8] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:50.093553] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 12169.00) [2024-09-26 16:00:50.093565] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:50.093581] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:50.093586] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:50.109771] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:50.109781] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.140450] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:50.140472] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.141266] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:50.141277] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337650141262) [2024-09-26 16:00:50.141283] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337650040898, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:50.141310] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223379376933, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.141317] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:50.141332] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:50.141337] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:50.141340] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:50.141347] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.141349] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.141351] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337650141286) [2024-09-26 16:00:50.170531] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:50.170540] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.171672] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337649826995]}) [2024-09-26 16:00:50.171734] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D912-0-0] [lt=7][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337650171664}) [2024-09-26 16:00:50.172716] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:50.200005] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=2] ====== check clog disk timer task ====== [2024-09-26 16:00:50.200015] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=8] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:50.200025] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:50.200962] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:50.200970] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.231246] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:50.231253] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.242008] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:50.242023] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337650242004) [2024-09-26 16:00:50.242028] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337650141286, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:50.242044] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223379477482, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.242049] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:50.242057] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:50.242062] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:50.242065] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:50.242070] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.242072] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.242074] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337650242031) [2024-09-26 16:00:50.247192] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589A-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:50.247231] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589A-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:50.261523] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:50.261532] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.272274] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337649826995]}) [2024-09-26 16:00:50.279701] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:50.279817] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=53, refresh_score_time=26, wash_time=2) [2024-09-26 16:00:50.280423] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1069538, clean_num=31457) [2024-09-26 16:00:50.291787] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:50.291797] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.309381] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=12][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337650239409, now=[mts=1727337649309369]) [2024-09-26 16:00:50.309417] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=36][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337650239409, now=[mts=1727337649309369], now0=[mts=1727337649309369], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:50.309424] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926915f700, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337649309227, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:50.309441] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f9263676038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915f700}) [2024-09-26 16:00:50.309449] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926915f700, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337649309227, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:50.309457] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926915f700, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337649309227, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337648309660, use_das=false, nested_level=0, session={this:0x7f9263676038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915f700}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:50.309473] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:50.309478] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:50.309481] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:50.309483] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:50.309485] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:50.309488] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:50.309493] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4] already timeout, do not need sleep(sleep_us=-84, remain_us=-84, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, timeout_timestamp=1727337650309409) [2024-09-26 16:00:50.309496] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:50.309504] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=5][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:50.309507] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:50.309508] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:50.309514] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:50.309565] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1764) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4012] get gts timeout(ret=-4012, expire_ts=1727337650239409, current_time=1727337650309377) [2024-09-26 16:00:50.309571] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=6][errcode=-4012] acquire global snapshot fail(ret=-4012, gts_ahead=0, expire_ts=1727337650239409, now=[mts=1727337650309564], now0=[mts=1727337650309564], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:50.309574] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4012] acquire global snapshot fail(ret=-4012, tx={this:0x7f926915f700, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337650309377, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:50.309581] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=7][errcode=-4012] fail to get snapshot(ret=-4012, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f9263676038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915f700}) [2024-09-26 16:00:50.309585] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4012] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4012, session_id=1, *tx_desc={this:0x7f926915f700, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337650309377, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:50.309591] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=6] start stmt(ret=-4012, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926915f700, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337650309377, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337648309660, use_das=false, nested_level=0, session={this:0x7f9263676038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926915f700}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:50.309601] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=10][errcode=-4012] fail to start stmt(ret=-4012, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:50.309604] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4012] fail start stmt(ret=-4012) [2024-09-26 16:00:50.309606] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4012] execute plan failed(ret=-4012) [2024-09-26 16:00:50.309610] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4012] open result set failed(ret=-4012) [2024-09-26 16:00:50.309611] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=1][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:50.309614] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:50.309616] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:50.309619] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:50.309620] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=1][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:50.309622] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-09-26 16:00:50.309626] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:50.309629] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1999969) [2024-09-26 16:00:50.309632] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:50.309637] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:50.309640] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:50.309643] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:50.309645] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:50.309650] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:50.309673] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:50.309675] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:50.309680] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=4][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:50.309683] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:50.309686] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:50.309689] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:50.309796] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C7F-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:50.309803] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:50.309806] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:50.309819] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:50.309837] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=17][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:50.310108] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:50.310116] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=8][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:50.322146] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:50.322153] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.328058] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.328069] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337650328055], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:50.328075] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337650328055], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:50.328079] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337650328055]}) [2024-09-26 16:00:50.342602] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:50.342612] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337650342599) [2024-09-26 16:00:50.342617] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337650242031, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:50.342633] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223379579085, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.342639] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:50.342645] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:50.342650] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:50.342652] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:50.342658] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.342660] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.342661] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337650342620) [2024-09-26 16:00:50.352373] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:50.352378] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.372712] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337650328055]}) [2024-09-26 16:00:50.372859] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:50.382625] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:50.382635] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.388448] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:50.388462] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=3] current monitor number(seq_=-1) [2024-09-26 16:00:50.396372] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D3E5F-0-0] [lt=11][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:50.398252] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1727337650398251, bytes: 14668810, bw: 0.003350 MB/s, add_ts: 1007803, add_bytes: 3540 [2024-09-26 16:00:50.412926] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:50.412933] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.442625] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:50.442638] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337650442622) [2024-09-26 16:00:50.442643] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337650342620, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:50.442661] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223379678680, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.442668] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:50.442684] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:50.442689] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:50.442692] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:50.442698] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.442700] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.442701] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337650442646) [2024-09-26 16:00:50.443153] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:50.443160] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.460165] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=15] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.460177] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.460183] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.460186] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.460194] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.460199] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.460218] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.460222] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.460224] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.460766] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.460772] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.460775] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.460779] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.460781] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.460783] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.461629] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.461637] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.461641] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.461656] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.461658] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.461662] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.461665] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.461667] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.461668] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.462306] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.462311] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.462313] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.462747] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.462755] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.462759] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.462763] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.462765] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.462767] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.462888] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.462891] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.462893] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.463699] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.463707] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.463710] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.463701] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.463715] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.463717] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.463716] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.463718] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.463719] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.464299] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.464309] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.464313] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:50.464317] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.464674] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.465031] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.465047] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.465374] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.465602] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.465609] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.465952] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.466572] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.466708] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.466719] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.467248] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.467371] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.467379] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.467910] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.467986] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.467995] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.468490] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.468568] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.468579] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.469374] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.469381] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.469982] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.469989] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.470086] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.470544] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.470551] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.470616] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.471513] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.471520] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.471547] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.472530] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.472536] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.472528] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.473473] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:50.473482] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.473504] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.473523] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.473521] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337650328055]}) [2024-09-26 16:00:50.474115] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.474137] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.474150] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.474940] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.474947] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.474959] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.476433] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.476440] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.476443] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.478402] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.478410] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.478417] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.479326] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.479332] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.479338] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.480366] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.480374] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.480402] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.480570] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=17] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:50.480669] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=50, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:50.481272] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.481278] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1100995, clean_num=31457) [2024-09-26 16:00:50.481290] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.481290] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.482284] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.482298] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.482366] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.484229] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.484230] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.484229] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.485187] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.485209] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=25][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.486171] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.486183] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.486182] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.487138] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.487139] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.487145] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.488134] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.488154] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.488161] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.488184] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:97, tid:3001}]) [2024-09-26 16:00:50.489104] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.489121] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.489126] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.490073] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.490073] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.490084] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.491043] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.491056] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.491062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.492030] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:50.503798] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:50.503804] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.522731] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591D-0-0] [lt=34][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:50.522740] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591D-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:50.522748] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591D-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:50.522753] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591D-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:50.535017] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:50.535025] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.542807] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:50.542819] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337650542803) [2024-09-26 16:00:50.542825] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337650442646, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:50.542831] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:50.542841] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:50.542844] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:50.543158] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:50.543169] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:50.543174] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:50.543177] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:50.543181] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:50.543183] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:50.543186] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:50.543188] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:50.543190] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:50.543192] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:50.543194] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:50.543196] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:50.543199] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:50.543201] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:50.543207] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:50.543249] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=41][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:50.543252] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:50.543255] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:50.543257] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:50.543260] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:50.543263] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:50.543272] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=6][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:50.543278] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:50.543280] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:50.543292] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=11][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:50.543297] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:50.543301] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:50.543303] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885763-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:50.543306] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:50.543308] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:50.543310] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:50.543313] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337650543082, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:50.543317] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:50.543319] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:50.543360] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:50.543364] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337650543363, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=528, wlock_time=11, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:50.543371] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:50.543377] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:50.543393] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223379779744, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.543398] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:50.543404] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:50.543409] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:50.543412] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:50.543426] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.543429] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.543431] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337650543380) [2024-09-26 16:00:50.544827] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6521, mem_used=12478464) [2024-09-26 16:00:50.564356] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=5] gc stale ls task succ [2024-09-26 16:00:50.565278] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:50.565284] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.573103] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:50.574102] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337650328055]}) [2024-09-26 16:00:50.595610] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:50.595618] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.601416] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.601428] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.602398] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.602415] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.625849] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:50.625858] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.629740] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] ====== traversal_flush timer task ====== [2024-09-26 16:00:50.629766] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] add dag success(dag=0x7f9264c3a030, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6163825430799493735, dag_cnt=1, dag_type_cnts=1) [2024-09-26 16:00:50.629773] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] success to create and add dag(ret=0, dag=0x7f9264c3a030) [2024-09-26 16:00:50.629776] INFO [STORAGE] flush (ob_tx_data_memtable.cpp:917) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] [TX DATA MERGE]schedule flush tx data memtable task done(ret=0, ret="OB_SUCCESS", get_ls_id()={id:1}, this=0x7f92a2294080, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:4, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337650629776, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:50.629811] INFO [MDS] flush (mds_table_mgr.cpp:172) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=19] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:50.629848] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=23] add dag success(dag=0x7f9264c3a560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6659188001345271846, dag_cnt=2, dag_type_cnts=1) [2024-09-26 16:00:50.629851] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c3a560) [2024-09-26 16:00:50.629855] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, create_flush_dag_time:1727337650629855, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452867, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200240}, checkpoint_unit_ptr:0x7f92a1a70850}, is_freeze_clock:true}) [2024-09-26 16:00:50.629863] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337650629842, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.629879] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=14] add dag success(dag=0x7f9264c3aa90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-9069613569758001279, dag_cnt=3, dag_type_cnts=2) [2024-09-26 16:00:50.629882] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c3aa90) [2024-09-26 16:00:50.629884] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, create_flush_dag_time:1727337650629883, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452892, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200489}, checkpoint_unit_ptr:0x7f92a1a70080}, is_freeze_clock:true}) [2024-09-26 16:00:50.629887] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337650629877, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.629897] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c3afc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1288986994123435108, dag_cnt=4, dag_type_cnts=3) [2024-09-26 16:00:50.629899] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3afc0) [2024-09-26 16:00:50.629901] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337650629901, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452911, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847222}, checkpoint_unit_ptr:0x7f92a1a71020}, is_freeze_clock:true}) [2024-09-26 16:00:50.629905] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337650629895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.629914] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3b4f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5355160120942094702, dag_cnt=5, dag_type_cnts=4) [2024-09-26 16:00:50.629916] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3b4f0) [2024-09-26 16:00:50.629918] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337650629918, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452929, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847223}, checkpoint_unit_ptr:0x7f92a1a717f0}, is_freeze_clock:true}) [2024-09-26 16:00:50.629921] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337650629912, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.629931] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405042) [2024-09-26 16:00:50.629942] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:50.629946] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405070) [2024-09-26 16:00:50.629953] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:50.629956] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405087) [2024-09-26 16:00:50.629963] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:50.629965] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405104) [2024-09-26 16:00:50.629972] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:50.629976] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] add dag success(dag=0x7f9264c3ba20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1994541446894784994, dag_cnt=6, dag_type_cnts=5) [2024-09-26 16:00:50.629979] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3ba20) [2024-09-26 16:00:50.629980] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, create_flush_dag_time:1727337650629980, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209556884, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:226}, checkpoint_unit_ptr:0x7f92a1a73f00}, is_freeze_clock:true}) [2024-09-26 16:00:50.629984] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337650629974, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.629997] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3bf50, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8791616776033068452, dag_cnt=7, dag_type_cnts=6) [2024-09-26 16:00:50.629999] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3bf50) [2024-09-26 16:00:50.630000] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337650630000, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405141, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847408}, checkpoint_unit_ptr:0x7f92a1a746d0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630004] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337650629991, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630014] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3c480, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6550230011160095889, dag_cnt=8, dag_type_cnts=7) [2024-09-26 16:00:50.630016] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3c480) [2024-09-26 16:00:50.630018] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337650630018, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405158, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847409}, checkpoint_unit_ptr:0x7f92a1a75670}, is_freeze_clock:true}) [2024-09-26 16:00:50.630021] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337650630011, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630030] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3c9b0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6809498982591245614, dag_cnt=9, dag_type_cnts=8) [2024-09-26 16:00:50.630032] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3c9b0) [2024-09-26 16:00:50.630034] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, create_flush_dag_time:1727337650630034, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405174, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:200004}, checkpoint_unit_ptr:0x7f92a1a74ea0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630037] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337650630028, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630046] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3cee0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6815944285528637599, dag_cnt=10, dag_type_cnts=9) [2024-09-26 16:00:50.630049] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c3cee0) [2024-09-26 16:00:50.630051] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, create_flush_dag_time:1727337650630051, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209506214, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1}, checkpoint_unit_ptr:0x7f92a1a76610}, is_freeze_clock:true}) [2024-09-26 16:00:50.630054] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337650630044, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630063] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3d410, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-4182959058374342994, dag_cnt=11, dag_type_cnts=10) [2024-09-26 16:00:50.630065] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3d410) [2024-09-26 16:00:50.630066] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, create_flush_dag_time:1727337650630066, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405204, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:412}, checkpoint_unit_ptr:0x7f92a1a75e40}, is_freeze_clock:true}) [2024-09-26 16:00:50.630070] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337650630061, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630078] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c3d940, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1586518267935340, dag_cnt=12, dag_type_cnts=11) [2024-09-26 16:00:50.630081] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c3d940) [2024-09-26 16:00:50.630082] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, create_flush_dag_time:1727337650630082, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405220, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:444}, checkpoint_unit_ptr:0x7f92a1a775b0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630086] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337650630076, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630097] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c3de70, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3044367941103776476, dag_cnt=13, dag_type_cnts=12) [2024-09-26 16:00:50.630099] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3de70) [2024-09-26 16:00:50.630101] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, create_flush_dag_time:1727337650630101, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455964, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:309}, checkpoint_unit_ptr:0x7f92a1a76de0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630104] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337650630094, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630113] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3e3a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1596830851104282047, dag_cnt=14, dag_type_cnts=13) [2024-09-26 16:00:50.630115] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3e3a0) [2024-09-26 16:00:50.630116] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, create_flush_dag_time:1727337650630116, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455986, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:331}, checkpoint_unit_ptr:0x7f92a1a77d80}, is_freeze_clock:true}) [2024-09-26 16:00:50.630120] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337650630111, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630131] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c3e8d0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2360195962081676673, dag_cnt=15, dag_type_cnts=14) [2024-09-26 16:00:50.630133] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3e8d0) [2024-09-26 16:00:50.630134] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, create_flush_dag_time:1727337650630134, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405262, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:458}, checkpoint_unit_ptr:0x7f92a1a7a490}, is_freeze_clock:true}) [2024-09-26 16:00:50.630138] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337650630128, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630147] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3ee00, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5675615614797020460, dag_cnt=16, dag_type_cnts=15) [2024-09-26 16:00:50.630149] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3ee00) [2024-09-26 16:00:50.630150] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, create_flush_dag_time:1727337650630150, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405279, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:487}, checkpoint_unit_ptr:0x7f92a1a79cc0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630154] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337650630145, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630163] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3f330, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-612435342908088006, dag_cnt=17, dag_type_cnts=16) [2024-09-26 16:00:50.630165] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3f330) [2024-09-26 16:00:50.630166] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, create_flush_dag_time:1727337650630166, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405296, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:455}, checkpoint_unit_ptr:0x7f92a1a794f0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630170] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337650630161, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630178] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3f860, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7868818205990873505, dag_cnt=18, dag_type_cnts=17) [2024-09-26 16:00:50.630180] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3f860) [2024-09-26 16:00:50.630184] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, create_flush_dag_time:1727337650630184, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405313, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:492}, checkpoint_unit_ptr:0x7f92a1a78550}, is_freeze_clock:true}) [2024-09-26 16:00:50.630188] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337650630177, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630197] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3fd90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7071731214611448811, dag_cnt=19, dag_type_cnts=18) [2024-09-26 16:00:50.630200] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c3fd90) [2024-09-26 16:00:50.630201] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, create_flush_dag_time:1727337650630201, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405347, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:501}, checkpoint_unit_ptr:0x7f92a1a78d20}, is_freeze_clock:true}) [2024-09-26 16:00:50.630204] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337650630195, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630213] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c402c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8731750006000903454, dag_cnt=20, dag_type_cnts=19) [2024-09-26 16:00:50.630215] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c402c0) [2024-09-26 16:00:50.630217] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, create_flush_dag_time:1727337650630217, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405365, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:472}, checkpoint_unit_ptr:0x7f92a1a7ac60}, is_freeze_clock:true}) [2024-09-26 16:00:50.630220] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337650630211, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630229] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c407f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6766388540859538791, dag_cnt=21, dag_type_cnts=20) [2024-09-26 16:00:50.630231] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c407f0) [2024-09-26 16:00:50.630232] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, create_flush_dag_time:1727337650630232, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405384, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:342}, checkpoint_unit_ptr:0x7f92a1a7cba0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630236] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337650630227, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630244] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c40d20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2986199386776543176, dag_cnt=22, dag_type_cnts=21) [2024-09-26 16:00:50.630247] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c40d20) [2024-09-26 16:00:50.630248] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337650630248, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405418, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:140}, checkpoint_unit_ptr:0x7f92a1a7b430}, is_freeze_clock:true}) [2024-09-26 16:00:50.630252] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337650630243, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630260] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c41250, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6947063304478012270, dag_cnt=23, dag_type_cnts=22) [2024-09-26 16:00:50.630263] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c41250) [2024-09-26 16:00:50.630264] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337650630264, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405436, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101009}, checkpoint_unit_ptr:0x7f92a1a7bc00}, is_freeze_clock:true}) [2024-09-26 16:00:50.630268] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337650630258, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630276] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c41780, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5843378789243662202, dag_cnt=24, dag_type_cnts=23) [2024-09-26 16:00:50.630278] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c41780) [2024-09-26 16:00:50.630280] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337650630280, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405453, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101010}, checkpoint_unit_ptr:0x7f92a1a7c3d0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630283] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337650630274, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630294] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c41cb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1841823684513746176, dag_cnt=25, dag_type_cnts=24) [2024-09-26 16:00:50.630296] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c41cb0) [2024-09-26 16:00:50.630298] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, create_flush_dag_time:1727337650630298, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405470, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:323}, checkpoint_unit_ptr:0x7f92a1a7d370}, is_freeze_clock:true}) [2024-09-26 16:00:50.630301] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337650630291, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630310] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c421e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-2678407261265885087, dag_cnt=26, dag_type_cnts=25) [2024-09-26 16:00:50.630312] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c421e0) [2024-09-26 16:00:50.630314] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, create_flush_dag_time:1727337650630314, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405487, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:377}, checkpoint_unit_ptr:0x7f92a1a7e310}, is_freeze_clock:true}) [2024-09-26 16:00:50.630317] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337650630308, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630328] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c42710, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7175089551615529855, dag_cnt=27, dag_type_cnts=26) [2024-09-26 16:00:50.630330] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c42710) [2024-09-26 16:00:50.630332] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, create_flush_dag_time:1727337650630332, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405503, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:297}, checkpoint_unit_ptr:0x7f92a1a7eae0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630335] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337650630325, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630344] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c42c40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3730526293242960203, dag_cnt=28, dag_type_cnts=27) [2024-09-26 16:00:50.630346] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c42c40) [2024-09-26 16:00:50.630348] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337650630348, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405519, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101065}, checkpoint_unit_ptr:0x7f92a1a7db40}, is_freeze_clock:true}) [2024-09-26 16:00:50.630351] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337650630342, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630362] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c43170, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=4663792628280492528, dag_cnt=29, dag_type_cnts=28) [2024-09-26 16:00:50.630364] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c43170) [2024-09-26 16:00:50.630366] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337650630366, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405536, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:333}, checkpoint_unit_ptr:0x7f92a3bc4850}, is_freeze_clock:true}) [2024-09-26 16:00:50.630369] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337650630359, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630378] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c436a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8399557287176188414, dag_cnt=30, dag_type_cnts=29) [2024-09-26 16:00:50.630380] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c436a0) [2024-09-26 16:00:50.630381] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337650630381, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405552, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101066}, checkpoint_unit_ptr:0x7f92a3bc5020}, is_freeze_clock:true}) [2024-09-26 16:00:50.630386] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337650630376, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630396] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c43bd0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8068938909778324349, dag_cnt=31, dag_type_cnts=30) [2024-09-26 16:00:50.630398] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c43bd0) [2024-09-26 16:00:50.630399] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, create_flush_dag_time:1727337650630399, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405569, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:334}, checkpoint_unit_ptr:0x7f92a3bc57f0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630403] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337650630393, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630412] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c44100, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7665372330263152188, dag_cnt=32, dag_type_cnts=31) [2024-09-26 16:00:50.630414] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c44100) [2024-09-26 16:00:50.630415] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, create_flush_dag_time:1727337650630415, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405586, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:451}, checkpoint_unit_ptr:0x7f92a3bc4080}, is_freeze_clock:true}) [2024-09-26 16:00:50.630420] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337650630410, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630430] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c44630, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6759548225246904317, dag_cnt=33, dag_type_cnts=32) [2024-09-26 16:00:50.630432] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c44630) [2024-09-26 16:00:50.630433] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, create_flush_dag_time:1727337650630433, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405602, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:378}, checkpoint_unit_ptr:0x7f92a1a7f2b0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630437] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337650630427, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630446] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c44b60, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8944614441376604790, dag_cnt=34, dag_type_cnts=33) [2024-09-26 16:00:50.630449] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c44b60) [2024-09-26 16:00:50.630451] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, create_flush_dag_time:1727337650630451, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209456003, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:121}, checkpoint_unit_ptr:0x7f92a3bc5fc0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630455] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337650630444, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630464] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c45090, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=507863347806783454, dag_cnt=35, dag_type_cnts=34) [2024-09-26 16:00:50.630466] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c45090) [2024-09-26 16:00:50.630468] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, create_flush_dag_time:1727337650630468, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405633, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:215}, checkpoint_unit_ptr:0x7f92a3bc86d0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630471] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337650630461, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630481] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c455c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1604282355669434429, dag_cnt=36, dag_type_cnts=35) [2024-09-26 16:00:50.630484] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c455c0) [2024-09-26 16:00:50.630485] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, create_flush_dag_time:1727337650630485, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405649, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:456}, checkpoint_unit_ptr:0x7f92a3bc6790}, is_freeze_clock:true}) [2024-09-26 16:00:50.630489] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337650630478, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630498] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c45af0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3553649416163308074, dag_cnt=37, dag_type_cnts=36) [2024-09-26 16:00:50.630500] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c45af0) [2024-09-26 16:00:50.630501] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337650630501, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405666, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:154}, checkpoint_unit_ptr:0x7f92a3bc6f60}, is_freeze_clock:true}) [2024-09-26 16:00:50.630505] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337650630496, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630515] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c46020, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8097406237500329300, dag_cnt=38, dag_type_cnts=37) [2024-09-26 16:00:50.630518] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c46020) [2024-09-26 16:00:50.630519] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337650630519, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405682, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101015}, checkpoint_unit_ptr:0x7f92a3bc7730}, is_freeze_clock:true}) [2024-09-26 16:00:50.630523] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337650630513, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630532] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c46550, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3976350754940852513, dag_cnt=39, dag_type_cnts=38) [2024-09-26 16:00:50.630534] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c46550) [2024-09-26 16:00:50.630535] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337650630535, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405699, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101016}, checkpoint_unit_ptr:0x7f92a3bc7f00}, is_freeze_clock:true}) [2024-09-26 16:00:50.630539] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337650630530, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630549] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c46a80, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8720304653345806796, dag_cnt=40, dag_type_cnts=39) [2024-09-26 16:00:50.630551] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c46a80) [2024-09-26 16:00:50.630553] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, create_flush_dag_time:1727337650630553, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405716, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101063}, checkpoint_unit_ptr:0x7f92a3bca610}, is_freeze_clock:true}) [2024-09-26 16:00:50.630556] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337650630547, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630565] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c46fb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1830562711513853292, dag_cnt=41, dag_type_cnts=40) [2024-09-26 16:00:50.630567] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c46fb0) [2024-09-26 16:00:50.630569] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, create_flush_dag_time:1727337650630569, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405733, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:452}, checkpoint_unit_ptr:0x7f92a3bc9e40}, is_freeze_clock:true}) [2024-09-26 16:00:50.630572] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337650630563, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630583] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c474e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1536604874905918360, dag_cnt=42, dag_type_cnts=41) [2024-09-26 16:00:50.630585] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c474e0) [2024-09-26 16:00:50.630587] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337650630587, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405750, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:328}, checkpoint_unit_ptr:0x7f92a3bc8ea0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630590] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337650630580, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630599] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c47a10, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6933628521298738662, dag_cnt=43, dag_type_cnts=42) [2024-09-26 16:00:50.630601] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c47a10) [2024-09-26 16:00:50.630603] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337650630603, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405767, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:330}, checkpoint_unit_ptr:0x7f92a3bc9670}, is_freeze_clock:true}) [2024-09-26 16:00:50.630607] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337650630597, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630617] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c47f40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1056199536334423564, dag_cnt=44, dag_type_cnts=43) [2024-09-26 16:00:50.630619] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c47f40) [2024-09-26 16:00:50.630621] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337650630621, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405784, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:329}, checkpoint_unit_ptr:0x7f92a3bcb5b0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630624] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337650630614, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630633] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c48470, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7405463607918801432, dag_cnt=45, dag_type_cnts=44) [2024-09-26 16:00:50.630635] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c48470) [2024-09-26 16:00:50.630637] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, create_flush_dag_time:1727337650630637, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405801, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:332}, checkpoint_unit_ptr:0x7f92a3bcade0}, is_freeze_clock:true}) [2024-09-26 16:00:50.630641] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337650630631, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630651] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c489a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6699723950958743903, dag_cnt=46, dag_type_cnts=45) [2024-09-26 16:00:50.630664] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=13] success to create and add dag(ret=0, dag=0x7f9264c489a0) [2024-09-26 16:00:50.630667] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, create_flush_dag_time:1727337650630667, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405818, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101067}, checkpoint_unit_ptr:0x7f92a3bcbd80}, is_freeze_clock:true}) [2024-09-26 16:00:50.630671] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337650630648, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.630678] INFO [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:655) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] traversal_flush successfully(ls_->get_ls_id()={id:1}, flush_tasks=[{table:0x7f92a1a70850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a70080, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71020, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a717f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71fc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72790, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72f60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73730, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73f00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a746d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75670, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a74ea0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76610, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75e40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a775b0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76de0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a77d80, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7a490, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a79cc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a794f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78550, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78d20, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7ac60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7cba0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7b430, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7bc00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7c3d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7d370, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7e310, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7eae0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7db40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a3bc4850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}]) [2024-09-26 16:00:50.630725] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=45] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-09-26 16:00:50.630747] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1961) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(dag_cnt=46, map_size=46) [2024-09-26 16:00:50.630751] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1971) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(running_dag_net_map_size=0, blocking_dag_net_list_size=0) [2024-09-26 16:00:50.630754] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(priority="PRIO_COMPACTION_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=46, waiting_dag_count=0) [2024-09-26 16:00:50.630757] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(priority="PRIO_HA_HIGH", low_limit=8, up_limit=8, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:50.630759] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_COMPACTION_MID", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:50.630760] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(priority="PRIO_HA_MID", low_limit=5, up_limit=5, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:50.630762] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_COMPACTION_LOW", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:50.630764] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_HA_LOW", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:50.630766] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_DDL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:50.630768] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(priority="PRIO_DDL_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:50.630769] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(priority="PRIO_TTL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:00:50.630771] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=45) [2024-09-26 16:00:50.630774] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=182) [2024-09-26 16:00:50.630776] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:50.630778] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:00:50.630780] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:50.630782] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:00:50.630784] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=1) [2024-09-26 16:00:50.630786] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=2) [2024-09-26 16:00:50.630787] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:50.630789] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:00:50.630790] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:00:50.630792] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:00:50.630794] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:00:50.630796] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:00:50.630798] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:00:50.630799] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:00:50.630801] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:00:50.630803] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:00:50.630805] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:00:50.630806] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:00:50.630808] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630810] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630811] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630815] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630817] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630819] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630820] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630821] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630823] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630824] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630826] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630828] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630829] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630831] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630833] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630835] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630836] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630837] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630839] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630841] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630842] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630844] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630846] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630849] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630851] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630853] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630854] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630856] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630857] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:00:50.630858] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630860] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-09-26 16:00:50.630861] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-09-26 16:00:50.630863] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-09-26 16:00:50.630865] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-09-26 16:00:50.630866] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:50.630868] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:50.630870] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:50.630872] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:50.630873] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:50.630875] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:50.630877] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:50.630880] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:50.630882] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:50.630884] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:50.630885] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:50.630887] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:50.630888] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:50.630890] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:50.630891] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:50.630893] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:50.630894] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:00:50.630895] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:00:50.630897] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:50.630898] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630900] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:50.630902] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630903] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:50.630905] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630907] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:50.630910] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630912] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:50.630914] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630915] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:50.630916] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630918] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:50.630919] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630921] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:50.630922] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630924] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:50.630926] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630928] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:00:50.630929] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:00:50.630932] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0) [2024-09-26 16:00:50.630934] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, scheduled_task_count=0) [2024-09-26 16:00:50.630936] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0) [2024-09-26 16:00:50.630937] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, scheduled_task_count=0) [2024-09-26 16:00:50.630939] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0) [2024-09-26 16:00:50.630940] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-09-26 16:00:50.630941] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0) [2024-09-26 16:00:50.630943] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-09-26 16:00:50.630945] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0) [2024-09-26 16:00:50.630947] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, scheduled_task_count=0) [2024-09-26 16:00:50.630949] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_net_count=0) [2024-09-26 16:00:50.630951] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_net_count=0) [2024-09-26 16:00:50.630954] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_net_count=0) [2024-09-26 16:00:50.630955] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_net_count=0) [2024-09-26 16:00:50.630957] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_net_count=0) [2024-09-26 16:00:50.630958] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_net_count=0) [2024-09-26 16:00:50.630959] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_net_count=0) [2024-09-26 16:00:50.630960] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_net_count=0) [2024-09-26 16:00:50.630962] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1991) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=184) [2024-09-26 16:00:50.630969] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337650630965, task_id:YB42C0A8002D-0006230056D8C6A0-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.630975] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=5] schedule one task(task=0x7f9264c3a3f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.630981] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337650630980, task_id:YB42C0A8002D-0006230056D8C6A1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.630984] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=3] schedule one task(task=0x7f9264c3a920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.630991] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337650630991, task_id:YB42C0A8002D-0006230056D8C6A2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.630994] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=2] schedule one task(task=0x7f9264c3ae50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.630999] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337650630998, task_id:YB42C0A8002D-0006230056D8C6A3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.631001] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=2] schedule one task(task=0x7f9264c3b380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.631008] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337650631007, task_id:YB42C0A8002D-0006230056D8C6A4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.631010] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=2] schedule one task(task=0x7f9264c3b8b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.631014] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650631013, task_id:YB42C0A8002D-0006230056D8C6A5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.631016] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=2] schedule one task(task=0x7f9264c3bde0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.631035] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.631034] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.631034] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.631039] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.631040] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.631040] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.631042] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726157244792838000) [2024-09-26 16:00:50.631043] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:50.631043] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:50.631045] INFO [STORAGE] ready_for_flush (ob_tx_data_memtable.cpp:865) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=2] memtable is frozen yet.(this=0x7f92a2294080) [2024-09-26 16:00:50.631051] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.631054] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.631080] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.631083] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.631085] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:50.631091] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.631096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:50.631109] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=6] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.631129] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.631132] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.631134] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:50.631139] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.631142] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847222}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:50.631142] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.631145] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.631146] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.631146] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.631150] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.631175] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=23] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:00:50.631184] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241777443612000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.639020] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=5] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:50.639035] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=9] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.641765] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.641792] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=26] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337650629912, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:50.641802] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.641898] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=14] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:50.641916] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=17] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.641928] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847223}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:50.641953] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:1152921504606847223}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847223}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650631077, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=10.85ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.642232] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=27] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:50.642878] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:50.642939] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:50.642948] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337650642937) [2024-09-26 16:00:50.642953] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337650542828, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:50.642967] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223379879370, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.642972] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:50.642979] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:50.642984] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:50.642982] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=6] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=7999, total_rows=55993) [2024-09-26 16:00:50.642986] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:50.642991] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.642993] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.642995] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337650642956) [2024-09-26 16:00:50.643082] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.643100] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=17] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337650629895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:50.643108] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.643170] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1, storage_schema:{this:0x7f9287db52b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.643193] INFO [STORAGE] prepare_tx_data_list (ob_tx_data_memtable.cpp:266) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=12] construct list more than once. skip this time(this=0x7f92a2294080) [2024-09-26 16:00:50.643222] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=12] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:50.643228] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.643237] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847222}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:50.643262] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=18] succeed to build merge ctx(tablet_id={id:1152921504606847222}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847222}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650631127, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=12.11ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.643485] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A4-0-0, dag_ret:0, dag_status:2, start_time:1727337650631010, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db9470, output_block_cnt_arr:0x7f9287db9478, concurrent_cnt:1, estimate_row_cnt:332829, estimate_occupy_size:150994944, latest_update_ts:1727337650631077, estimated_finish_time:1727337667643539}) [2024-09-26 16:00:50.643498] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=12] succeed to init merge ctx(task={this:0x7f9264c3b8b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A4-0-0, dag_ret:0, dag_status:2, start_time:1727337650631010, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.643506] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=7] task finish process(ret=0, start_time=1727337650631075, end_time=1727337650643505, runtime=12430, *this={this:0x7f9264c3b8b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A4-0-0, dag_ret:0, dag_status:2, start_time:1727337650631010, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.643540] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=2] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.643557] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=25] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:50.643612] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92a9b52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d2e060, count:4, allocator:0x7f92b33fe7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d2e350, local_datums:0x7f9287d2e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:50.643635] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=20] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92a9b52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265a74060, count:4, allocator:0x7f92b33ff558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9265a74350, local_datums:0x7f9265a74350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:50.643884] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:318) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=7] successfully alloc block(macro_id=[1091](ver=0,mode=0,seq=824628)) [2024-09-26 16:00:50.644347] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=6] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=8366, total_rows=58562) [2024-09-26 16:00:50.644772] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A3-0-0, dag_ret:0, dag_status:2, start_time:1727337650631001, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1470, output_block_cnt_arr:0x7f92728f1478, concurrent_cnt:1, estimate_row_cnt:329211, estimate_occupy_size:150994944, latest_update_ts:1727337650631127, estimated_finish_time:1727337667644844}) [2024-09-26 16:00:50.644786] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f9264c3b380, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A3-0-0, dag_ret:0, dag_status:2, start_time:1727337650631001, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.644796] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=9] task finish process(ret=0, start_time=1727337650631125, end_time=1727337650644796, runtime=13671, *this={this:0x7f9264c3b380, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A3-0-0, dag_ret:0, dag_status:2, start_time:1727337650631001, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.644851] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.644864] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=13] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337650629877, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=17, max_schema_version_on_memtable=1725120002010696) [2024-09-26 16:00:50.644871] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=17, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.644975] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1887, total_rows=92463) [2024-09-26 16:00:50.644979] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.644987] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200489}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:19, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.645007] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:200489}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200489}, table_count:2, [{i:0, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650631026, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=13.96ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.645289] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A2-0-0, dag_ret:0, dag_status:2, start_time:1727337650630993, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505460, output_block_cnt_arr:0x7f9266505468, concurrent_cnt:1, estimate_row_cnt:254187, estimate_occupy_size:138412032, latest_update_ts:1727337650631026, estimated_finish_time:1727337667645355}) [2024-09-26 16:00:50.645302] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f9264c3ae50, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A2-0-0, dag_ret:0, dag_status:2, start_time:1727337650630993, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.645319] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=16] task finish process(ret=0, start_time=1727337650631021, end_time=1727337650645319, runtime=14298, *this={this:0x7f9264c3ae50, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A2-0-0, dag_ret:0, dag_status:2, start_time:1727337650630993, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.645759] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337650629974, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.645777] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=17] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752755184, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828752755184, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.645859] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=76] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.645863] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.645870] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:226}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752755184, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.645890] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:226}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752755184, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:226}, table_count:1, [{i:0, table_key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}, merge_scn:{val:1726241777443612000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650631138, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=14.73ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.645915] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A5-0-0, dag_ret:0, dag_status:2, start_time:1727337650631016, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3480, output_block_cnt_arr:0x7f92728f3488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:56623104, latest_update_ts:1727337650631138, estimated_finish_time:1727337667645942}) [2024-09-26 16:00:50.645923] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3bde0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A5-0-0, dag_ret:0, dag_status:2, start_time:1727337650631016, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:50.645928] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=5] task finish process(ret=0, start_time=1727337650631136, end_time=1727337650645928, runtime=14792, *this={this:0x7f9264c3bde0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A5-0-0, dag_ret:0, dag_status:2, start_time:1727337650631016, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:50.646129] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, old_min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726159999077775000, old_tablet_snaphsot_version=1726163995122514000, tablet={this:0x7f92b8637b60, is_inited:true, wash_score:1727337650628844000, hold_ref_cnt:true, ref_cnt:2, version:2, length:503, tablet_addr:[814679-1621-0-2305843643796488192-2](offset=1376256,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:200240}, data_tablet_id:{id:200240}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726158221127945000, v:0}, clog_checkpoint_scn:{val:1726163995200406000, v:0}, ddl_checkpoint_scn:{val:1711609699643766000, v:0}, snapshot_version:1726163995122514000, multi_version_start:1726159999077775000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726158220122116000, cur_report_version:1726158220122116000, data_checksum:2675562562, row_count:146957}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:1711609699490656001, v:0}, ddl_snapshot_version:1711609699456885000, max_sync_storage_schema_version:1711609699311648, max_serialized_medium_scn:0, ddl_execution_id:1, ddl_data_format_version:17180000769, ddl_commit_scn:{val:1711609699643766000, v:0}, mds_checkpoint_scn:{val:1726163586417012001, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1711609699311648}, table_store_addr:{ptr:{this:0x7f92b86381f4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814679-1621-0-2305845774100242432-0](offset=1351680,size=2575,type=2,seq=0), type:"MAJOR", tablet_id:{id:200240}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726158220122116000, v:0}}, snapshot_version:1726158220122116000, max_merge_version:1726158220122116000}}, minor_tables:{{i:0, addr:[814679-1621-0-2305844370719681039-0](offset=1354255,size=1268,type=2,seq=0), type:"MINI", tablet_id:{id:200240}, scn_range:{start_scn:{val:1726158221127945000, v:0}, end_scn:{val:1726159107115548000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726159106450605000, upper_trans_version:1726159106450605000}, {i:1, addr:[814679-1621-0-2305846187490848515-0](offset=1355523,size=2960,type=2,seq=0), type:"MINOR", tablet_id:{id:200240}, scn_range:{start_scn:{val:1726159107115548000, v:0}, end_scn:{val:1726163187167245000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726163187058024000, upper_trans_version:1726163187058024000}, {i:2, addr:[814679-1621-0-2305844371793427091-0](offset=1358483,size=1269,type=2,seq=0), type:"MINI", tablet_id:{id:200240}, scn_range:{start_scn:{val:1726163187167245000, v:0}, end_scn:{val:1726163995200406000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726163995191699000, upper_trans_version:1726163995191699000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337650629842, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814679-1621-0-2305843593330610176-2](offset=1363968,size=544,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814679-1621-0-2305843163833881120-0](offset=1364512,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814679-1621-0-2305843255101936304-0](offset=1364656,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814679-1621-0-2305843235774583701-0](offset=1364885,size=211,type=2,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:1726158220122116000}, medium_info_list:{ptr:(nil), addr:[814679-1621-0-2305843463407857664-0](offset=1372160,size=423,type=2,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1711609699370224000, v:0}, create_commit_version:1711609699370224000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a70850, 0x7f92a1a71fc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.646189] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=59] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726158220122116000) [2024-09-26 16:00:50.646201] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.646227] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=5] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.646236] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=6] schedule one task(task=0x7f9264d76030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.646240] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=2] schedule one task(task=0x7f9287cf83c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.646328] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:4, allocator:0x7f92864107d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263b5a350, local_datums:0x7f9263b5a350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:50.646351] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=20] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:4, allocator:0x7f9286411558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263b5e350, local_datums:0x7f9263b5e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:50.646612] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:318) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=7] successfully alloc block(macro_id=[1812](ver=0,mode=0,seq=824629)) [2024-09-26 16:00:50.648301] INFO [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:329) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=4] pre-process commit versions row(get_ls_id()={id:1}, do_recycle_=true, current_time=1727337650648300, prev_recycle_time=0) [2024-09-26 16:00:50.650514] INFO [STORAGE] get_ls_min_end_scn (ob_ls_tablet_service.cpp:5618) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=6] get ls min end scn finish(ls_checkpoint={val:1726163586617842000, v:0}) [2024-09-26 16:00:50.650527] WDIAG [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:761) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=9][errcode=-4023] failed to get snapshot version(ret=-4023, MTL_ID()=1) [2024-09-26 16:00:50.650531] INFO [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:773) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=3] get tx data recycle scn finish.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, recycle_scn={val:1726076389100953000, v:0}, min_end_scn_from_old_tablets={val:4611686018427387903, v:0}, min_end_scn_from_latest_tablets={val:1726076389100953000, v:0}) [2024-09-26 16:00:50.650537] WDIAG [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:336) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=5][errcode=-4023] get recycle ts failed.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:50.650541] WDIAG [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:236) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=2][errcode=-4023] process commit version row failed.(ret=-4023, ret="OB_EAGAIN", this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337650629776, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:50.650555] INFO [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:254) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=14] [TX DATA MERGE]pre process for merge done(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}, this=0x7f92a2294080, tg=time guard have no click events for optimization, fake_tx_data_guard={tx_data:{tx_id:{txid:9223372036854775807}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337650629776, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:50.650569] WDIAG [STORAGE.COMPACTION] pre_process_tx_data_table_merge_ (ob_tx_table_merge_task.cpp:117) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=13][errcode=-4023] do pre process for tx data table merge failed.(ret=-4023, ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, table={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337650629776, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:50.650579] WDIAG [STORAGE.COMPACTION] inner_init_ctx (ob_tx_table_merge_task.cpp:157) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=9][errcode=-4023] pre process tx data table for merge failed.(ret=-4023, ret="OB_EAGAIN", ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.650582] WDIAG [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1061) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=2][errcode=-4023] fail to inner init ctx(ret=-4023, tablet_id={id:49402}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, create_snapshot_version:0, is_full_merge:false, merge_level:1, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:4, range_array:[], concurrent_cnt:0, is_inited:false}, schema_ctx:{schema_version:1, storage_schema:{this:0x7f9287db52b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:49402}, table_count:1, [{i:0, table_key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref:4}]}, schedule_major:false, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}, merge_scn:{val:4611686018427387903, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:NULL, compaction_filter:NULL, time_guard:total=19.55ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}) [2024-09-26 16:00:50.650605] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:975) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=23][errcode=-4023] failed to build merge ctx(ret=-4023, ctx->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.650608] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:994) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=3][errcode=-4023] sstable merge finish(ret=-4023, ctx=0x7f9287db4060, task={this:0x7f9264c3a3f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C6A0-0-0, dag_ret:0, dag_status:2, start_time:1727337650630974, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:50.650616] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=8][errcode=-4023] failed to process task(ret=-4023) [2024-09-26 16:00:50.650619] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=3] task finish process(ret=-4023, start_time=1727337650631030, end_time=1727337650650619, runtime=19589, *this={this:0x7f9264c3a3f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C6A0-0-0, dag_ret:0, dag_status:2, start_time:1727337650630974, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:50.650626] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=6][errcode=-4023] failed to do work(ret=-4023, *task_={this:0x7f9264c3a3f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C6A0-0-0, dag_ret:0, dag_status:2, start_time:1727337650630974, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:50.650634] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=6] dag finished(dag_ret=-4023, runtime=19660, dag_cnt=45, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264c3a030, dag={ObIDag:{this:0x7f9264c3a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C6A0-0-0, dag_ret:-4023, dag_status:5, start_time:1727337650630974, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}) [2024-09-26 16:00:50.650645] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2871][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6A0-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650630965, task_id:YB42C0A8002D-0006230056D8C6A0-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.650729] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685ea060, count:3, allocator:0x7f9269f527d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92685ea320, local_datums:0x7f92685ea320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:50.650750] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=19] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fe060, count:3, allocator:0x7f9269f53558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fe320, local_datums:0x7f92665fe320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:50.650912] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=9][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:50.650978] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.650983] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.650991] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:50.650994] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:50.650996] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2070, hex: 9F094D0300000000, int: 55380383},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e20a8, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e20e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3044, column_count:19, rowkey_column_count:3, has_column_checksum:0, row_count:83, row_store_type:0, opt:5, var_column_count:0, row_offset:16109, original_length:16381, max_merged_trans_version:1726163995623997000, data_length:16381, data_zlength:7759, data_checksum:3416849541, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92665a6030, buf_size:7759, data_size:16381, row_count:83, column_count:19, max_merged_trans_version:1726163995623997000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3537734793, row_count_delta:83, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16381}) [2024-09-26 16:00:50.651012] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=16][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:50.651015] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=3][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:50.651016] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:50.651018] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:50.651020] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fe320, local_datums:0x7f92665fe320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fe4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fe6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fe320, local_datums:0x7f92665fe320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fe4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fe6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f9269f52e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269f52e98, limit_param:null, stmt_allocator:0x7f9269f53558, allocator:0x7f9269f53558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320384, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fe320, local_datums:0x7f92665fe320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fe4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fe6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337650629877, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}) [2024-09-26 16:00:50.651094] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=74][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:50.651096] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fe320, local_datums:0x7f92665fe320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fe4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fe6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f9269f52e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269f52e98, limit_param:null, stmt_allocator:0x7f9269f53558, allocator:0x7f9269f53558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320384, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fe320, local_datums:0x7f92665fe320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fe4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fe6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337650629877, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}]) [2024-09-26 16:00:50.651135] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=38][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:50.651137] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.651151] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=10] partition merge iter row count(i=1, row_count=83, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, table={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337650629877, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.651161] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:50.651183] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=11][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:50.651189] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=6][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:50.651191] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337650650675, end_time=1727337650651191, runtime=516, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A2-0-0, dag_ret:0, dag_status:2, start_time:1727337650630993, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.651200] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A2-0-0, dag_ret:0, dag_status:2, start_time:1727337650630993, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:50.651208] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=20215, dag_cnt=44, dag_cnts_[dag.get_type()]=44, &dag=0x7f9264c3aa90, dag={ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A2-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650630993, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.651218] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A2-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650630991, task_id:YB42C0A8002D-0006230056D8C6A2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.654423] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=226, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b08060, count:5, allocator:0x7f92864067d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9265b08360, local_datums:0x7f9265b08360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:50.657085] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650657080, task_id:YB42C0A8002D-0006230056D8C6A6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.657095] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=9] schedule one task(task=0x7f9264c3c310, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.657103] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337650657102, task_id:YB42C0A8002D-0006230056D8C6A7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.657106] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=3] schedule one task(task=0x7f9264c3c840, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.657137] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.657142] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.657146] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.657154] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.660844] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=19][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:50.660905] ERROR issue_dba_error (ob_log.cpp:1875) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=10][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.660909] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.660917] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:50.660920] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:50.660923] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=3][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd8070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd80a8, hex: 0000000000000000, int: 0},idx=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9287dd80e0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd8118, hex: 58A3C56A4A280BE8, int: -1726241732082425000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd8150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-19590, column_count:10, rowkey_column_count:5, has_column_checksum:0, row_count:224, row_store_type:0, opt:0, var_column_count:0, row_offset:15527, original_length:16363, max_merged_trans_version:1726241777443612000, data_length:16363, data_zlength:16363, data_checksum:2057046089, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926fc04090, buf_size:16363, data_size:16363, row_count:224, column_count:10, max_merged_trans_version:1726241777443612000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1006468742, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16363}) [2024-09-26 16:00:50.660953] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=30][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:50.660956] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=3][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:50.660958] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:50.660964] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=5][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:50.660966] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:50.660968] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:50.660970] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9265b08360, local_datums:0x7f9265b08360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9265b083e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9265b08360, local_datums:0x7f9265b08360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9265b083e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f9286406118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286406118, limit_param:null, stmt_allocator:0x7f92864067d8, allocator:0x7f92864067d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9265b08360, local_datums:0x7f9265b08360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9265b083e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337650629974, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}) [2024-09-26 16:00:50.661044] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=74][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:50.661046] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9265b08360, local_datums:0x7f9265b08360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9265b083e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f9286406118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286406118, limit_param:null, stmt_allocator:0x7f92864067d8, allocator:0x7f92864067d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9265b08360, local_datums:0x7f9265b08360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9265b083e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337650629974, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}]) [2024-09-26 16:00:50.661082] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=35][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:50.661084] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=1] partition merge iter row count(i=0, row_count=224, ghost_row_count=0, pkey={tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, table={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337650629974, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.661098] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:50.661111] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:50.661115] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:50.661117] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337650654355, end_time=1727337650661117, runtime=6762, *this={this:0x7f9287cf83c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A5-0-0, dag_ret:0, dag_status:2, start_time:1727337650631016, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:50.661124] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287cf83c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A5-0-0, dag_ret:0, dag_status:2, start_time:1727337650631016, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:50.661134] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=6] dag finished(dag_ret=-4184, runtime=30118, dag_cnt=43, dag_cnts_[dag.get_type()]=43, &dag=0x7f9264c3ba20, dag={ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A5-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650631016, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}) [2024-09-26 16:00:50.661144] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A5-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650631013, task_id:YB42C0A8002D-0006230056D8C6A5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.661283] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:50.661290] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.662009] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:50.662047] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.662060] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=13] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337650629842, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=18, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:50.662068] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.662257] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=12] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:50.662265] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=7] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.662273] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200240}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.662295] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:200240}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200240}, table_count:2, [{i:0, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8637b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650631026, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=31.24ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.662515] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=23] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:50.662937] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.662947] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=10][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.662952] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.662959] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.663015] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=38] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A1-0-0, dag_ret:0, dag_status:2, start_time:1727337650630984, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3460, output_block_cnt_arr:0x7f9287db3468, concurrent_cnt:1, estimate_row_cnt:85417, estimate_occupy_size:134217728, latest_update_ts:1727337650631026, estimated_finish_time:1727337667663078}) [2024-09-26 16:00:50.663027] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=12] succeed to init merge ctx(task={this:0x7f9264c3a920, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A1-0-0, dag_ret:0, dag_status:2, start_time:1727337650630984, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.663035] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=7] task finish process(ret=0, start_time=1727337650631023, end_time=1727337650663034, runtime=32011, *this={this:0x7f9264c3a920, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A1-0-0, dag_ret:0, dag_status:2, start_time:1727337650630984, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.663055] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=5] schedule one task(task=0x7f9264bb8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.663067] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337650663065, task_id:YB42C0A8002D-0006230056D8C6A8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.663072] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=5] schedule one task(task=0x7f9264c3cd70, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.663159] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=14] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92aae56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265894060, count:3, allocator:0x7f926369a7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9265894320, local_datums:0x7f9265894320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:50.663185] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=23] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92aae56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f927282e060, count:3, allocator:0x7f926369b558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f927282e320, local_datums:0x7f927282e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:50.663464] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=10][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:50.663518] ERROR issue_dba_error (ob_log.cpp:1875) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.663522] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf2041c8 0xf217c49 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.663545] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=22][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:50.663548] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:50.663550] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9265990108, hex: 313030303030333830, cstr: 100000380},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92659900a8, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92659900e0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-29413, column_count:20, rowkey_column_count:3, has_column_checksum:0, row_count:100, row_store_type:0, opt:0, var_column_count:0, row_offset:16082, original_length:16422, max_merged_trans_version:1726189672100745000, data_length:16454, data_zlength:5272, data_checksum:4171874375, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9265992030, buf_size:5272, data_size:16454, row_count:100, column_count:20, max_merged_trans_version:1726189672100745000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:371002229, row_count_delta:22, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16422}) [2024-09-26 16:00:50.663564] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=14][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:50.663566] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:50.663561] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:50.663568] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:50.663569] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:50.663579] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=8] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.663629] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8640e00, is_inited:true, wash_score:1727337650628844000, hold_ref_cnt:true, ref_cnt:2, version:2, length:506, tablet_addr:[816017-1849-0-2305843647017144320-1](offset=806912,size=594,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, data_tablet_id:{id:200489}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725991206240736000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1725991205062061000, cur_report_version:1725991205062061000, data_checksum:1029137533, row_count:46137450}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1725120000848360, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1725120000848360}, table_store_addr:{ptr:null, addr:[816017-1849-0-2305845308095737856-1](offset=798720,size=2141,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718219357-0](offset=800861,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255101372665-0](offset=801017,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1725991205062061000}, medium_info_list:{ptr:(nil), addr:[816017-1849-0-2305843505283219456-0](offset=802816,size=462,type=2,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1725120001569133000, v:0}, create_commit_version:1725120001569133000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a75670, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:50.663659] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=30] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:50.663666] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.663571] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1526) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, partition_fuser_->get_result_row()={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9265892148, local_datums:0x7f9265892148}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265892190, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92658921c8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92658923c0, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92658923f8, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, fuse_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9265894320, local_datums:0x7f9265894320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92658943a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92658945d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f926369a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926369a118, limit_param:null, stmt_allocator:0x7f926369a7d8, allocator:0x7f926369a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:83887108, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9265894320, local_datums:0x7f9265894320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92658943a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92658945d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f927282e320, local_datums:0x7f927282e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f926369ae98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926369ae98, limit_param:null, stmt_allocator:0x7f926369b558, allocator:0x7f926369b558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3760845666523558192, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f927282e320, local_datums:0x7f927282e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337650629842, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:50.663689] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.663693] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.663695] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.663699] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.663703] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.663707] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.663681] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=110][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9265894320, local_datums:0x7f9265894320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92658943a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92658945d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f926369a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926369a118, limit_param:null, stmt_allocator:0x7f926369a7d8, allocator:0x7f926369a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:83887108, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9265894320, local_datums:0x7f9265894320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92658943a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92658945d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f927282e320, local_datums:0x7f927282e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f926369ae98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926369ae98, limit_param:null, stmt_allocator:0x7f926369b558, allocator:0x7f926369b558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3760845666523558192, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f927282e320, local_datums:0x7f927282e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337650629842, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:50.663797] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=115][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:50.663800] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=2] partition merge iter row count(i=0, row_count=23, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.663813] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=9] partition merge iter row count(i=1, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337650629842, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.663822] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=6][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:50.663833] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:50.663836] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:50.663839] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=3] task finish process(ret=-4184, start_time=1727337650663104, end_time=1727337650663839, runtime=735, *this={this:0x7f9264bb8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A1-0-0, dag_ret:0, dag_status:2, start_time:1727337650630984, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.663848] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=9][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264bb8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A1-0-0, dag_ret:0, dag_status:2, start_time:1727337650630984, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:50.663857] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=32873, dag_cnt=42, dag_cnts_[dag.get_type()]=42, &dag=0x7f9264c3a560, dag={ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A1-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650630984, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.663896] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A1-0-0] [lt=36] succeed to del sys task(removed_task={start_time:1727337650630980, task_id:YB42C0A8002D-0006230056D8C6A1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.664032] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337650664028, task_id:YB42C0A8002D-0006230056D8C6A9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.664039] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=6] schedule one task(task=0x7f9264c3d2a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.664056] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337650630028, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.664069] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828761605424, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828761605424, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.664083] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.664087] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.664093] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200004}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1710828761605424, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.664111] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:200004}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828761605424, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200004}, table_count:1, [{i:0, table_key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8621da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650663683, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=407us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.664142] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=25] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A8-0-0, dag_ret:0, dag_status:2, start_time:1727337650663072, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5460, output_block_cnt_arr:0x7f9287db5468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337650663683, estimated_finish_time:1727337667664145}) [2024-09-26 16:00:50.664149] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3cd70, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A8-0-0, dag_ret:0, dag_status:2, start_time:1727337650663072, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.664155] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=6] task finish process(ret=0, start_time=1727337650663681, end_time=1727337650664155, runtime=474, *this={this:0x7f9264c3cd70, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A8-0-0, dag_ret:0, dag_status:2, start_time:1727337650663072, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.664171] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=4] schedule one task(task=0x7f9264bb8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.664365] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=200004, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265894060, count:3, allocator:0x7f92636887d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265894320, local_datums:0x7f9265894320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:50.664641] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=18][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:50.664710] ERROR issue_dba_error (ob_log.cpp:1875) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.664715] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.664723] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:50.664727] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=4][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:50.664730] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e0a8, hex: C852A9F488570BE8, int: -1726189786434547000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e0e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19071, column_count:8, rowkey_column_count:3, has_column_checksum:0, row_count:218, row_store_type:0, opt:0, var_column_count:0, row_offset:15549, original_length:16361, max_merged_trans_version:1726190661653507000, data_length:16361, data_zlength:5996, data_checksum:2604580500, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9272830030, buf_size:5996, data_size:16361, row_count:218, column_count:8, max_merged_trans_version:1726190661653507000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:83488362, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16361}) [2024-09-26 16:00:50.664744] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=13][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:50.664747] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=3][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:50.664749] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:50.664755] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=6][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:50.664758] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265894320, local_datums:0x7f9265894320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92658943a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265894320, local_datums:0x7f9265894320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92658943a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f9263688118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263688118, limit_param:null, stmt_allocator:0x7f92636887d8, allocator:0x7f92636887d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:83887108, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265894320, local_datums:0x7f9265894320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92658943a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337650630028, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:50.664862] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=104][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:50.664865] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=3][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265894320, local_datums:0x7f9265894320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92658943a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f9263688118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263688118, limit_param:null, stmt_allocator:0x7f92636887d8, allocator:0x7f92636887d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:83887108, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265894320, local_datums:0x7f9265894320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92658943a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9265894480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337650630028, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:50.664912] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=47][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:50.664915] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=2] partition merge iter row count(i=0, row_count=218, ghost_row_count=0, pkey={tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337650630028, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.664931] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=11][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:50.664942] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:50.664946] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:50.664949] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=3] task finish process(ret=-4184, start_time=1727337650664325, end_time=1727337650664949, runtime=624, *this={this:0x7f9264bb8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A8-0-0, dag_ret:0, dag_status:2, start_time:1727337650663072, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.664958] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264bb8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A8-0-0, dag_ret:0, dag_status:2, start_time:1727337650663072, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:50.664967] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=6] dag finished(dag_ret=-4184, runtime=1895, dag_cnt=41, dag_cnts_[dag.get_type()]=41, &dag=0x7f9264c3c9b0, dag={ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A8-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650663072, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.664977] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A8-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337650663065, task_id:YB42C0A8002D-0006230056D8C6A8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.665016] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.665020] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.665024] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.665030] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.665035] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861eec0, is_inited:true, wash_score:1727337650628844000, hold_ref_cnt:true, ref_cnt:2, version:2, length:493, tablet_addr:[816017-1849-0-2305843633059586048-2](offset=1892352,size=581,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1}, data_tablet_id:{id:1}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:778798285, row_count:7249}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861f59c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844357835309056-0](offset=1884160,size=1256,type=2,seq=0), type:"MAJOR", tablet_id:{id:1}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077591663360001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843964845802728-0](offset=1885416,size=890,type=2,seq=0), type:"MINOR", tablet_id:{id:1}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178578410947000, upper_trans_version:1726178578410947000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337650630044, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcd4f0, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241621541056000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405180}, this:0x7f92a3bcd4f0, timestamp:1727334209405180, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:53, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277807268534000, v:0}, rec_scn:{val:1726241631569302000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771609600-2](offset=1888256,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843169203114317-0](offset=1888589,size=149,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881234914-0](offset=1888738,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828751995331000, v:0}, create_commit_version:1710828751995331000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a76610, 0x7f92a3bcd4f0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.665084] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=49] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.665095] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241621541056000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.665199] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337650665197, task_id:YB42C0A8002D-0006230056D8C6AA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.665205] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=5] schedule one task(task=0x7f9264c3d7d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.665705] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337650630044, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.665726] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=20] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:0, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.665743] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.665747] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.665753] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:0, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.665773] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:1}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:0, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1}, table_count:1, [{i:0, table_key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}, merge_scn:{val:1726241621541056000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650665009, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=742us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.665828] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=38] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A9-0-0, dag_ret:0, dag_status:2, start_time:1727337650664039, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3480, output_block_cnt_arr:0x7f92728f3488, concurrent_cnt:1, estimate_row_cnt:3, estimate_occupy_size:6291456, latest_update_ts:1727337650665009, estimated_finish_time:1727337667665831}) [2024-09-26 16:00:50.665841] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=12] succeed to init merge ctx(task={this:0x7f9264c3d2a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A9-0-0, dag_ret:0, dag_status:2, start_time:1727337650664039, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:50.665848] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=6] task finish process(ret=0, start_time=1727337650665007, end_time=1727337650665847, runtime=840, *this={this:0x7f9264c3d2a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A9-0-0, dag_ret:0, dag_status:2, start_time:1727337650664039, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:50.665869] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=4] schedule one task(task=0x7f926592a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.665926] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=1, range=0x7f92aab52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:8}, cols_desc:{ObIArray:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b06060, count:5, allocator:0x7f92b323c7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b06360, local_datums:0x7f9265b06360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:50.666311] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:50.666357] ERROR issue_dba_error (ob_log.cpp:1875) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.666360] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.666367] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:50.666369] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:50.666371] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 17, flag: 0, null: 0, ptr: 0x7f9264bb0178, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264bb00a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 15, flag: 0, null: 0, ptr: 0x7f9264bb0189, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264bb0118, hex: 60D2640CCC290BE8, int: -1726240075808452000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264bb0150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-20032, column_count:8, rowkey_column_count:5, has_column_checksum:0, row_count:158, row_store_type:0, opt:0, var_column_count:0, row_offset:15849, original_length:16421, max_merged_trans_version:1726241621541056000, data_length:16421, data_zlength:16421, data_checksum:3599888316, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6c04090, buf_size:16421, data_size:16421, row_count:158, column_count:8, max_merged_trans_version:1726241621541056000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1714289658, row_count_delta:2, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16421}) [2024-09-26 16:00:50.666384] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=13][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:50.666386] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:50.666389] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=3][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:50.666390] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:50.666394] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=4][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:50.666396] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:50.666398] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b06360, local_datums:0x7f9265b06360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b063a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b06418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b06450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b06360, local_datums:0x7f9265b06360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b063a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b06418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b06450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f92b323c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b323c118, limit_param:null, stmt_allocator:0x7f92b323c7d8, allocator:0x7f92b323c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320133, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b06360, local_datums:0x7f9265b06360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b063a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b06418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b06450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337650630044, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:50.666470] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=72][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:50.666472] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b06360, local_datums:0x7f9265b06360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b063a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b06418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b06450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f92b323c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b323c118, limit_param:null, stmt_allocator:0x7f92b323c7d8, allocator:0x7f92b323c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320133, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b06360, local_datums:0x7f9265b06360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b063a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b06418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b06450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337650630044, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:50.666511] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=38][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:50.666513] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=1] partition merge iter row count(i=0, row_count=158, ghost_row_count=0, pkey={tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, table={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337650630044, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.666525] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:50.666536] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:50.666540] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:50.666542] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337650665883, end_time=1727337650666541, runtime=658, *this={this:0x7f926592a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A9-0-0, dag_ret:0, dag_status:2, start_time:1727337650664039, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:50.666548] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926592a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A9-0-0, dag_ret:0, dag_status:2, start_time:1727337650664039, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:50.666556] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=2517, dag_cnt=40, dag_cnts_[dag.get_type()]=40, &dag=0x7f9264c3cee0, dag={ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A9-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650664039, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}) [2024-09-26 16:00:50.666565] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A9-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650664028, task_id:YB42C0A8002D-0006230056D8C6A9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.666591] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337650666589, task_id:YB42C0A8002D-0006230056D8C6AB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.666595] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=3] schedule one task(task=0x7f9264c3dd00, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.666643] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.666650] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.666653] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.666658] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.666662] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.666671] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.666985] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337650629991, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.667003] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=17] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:50.667007] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=4] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9266505480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.667124] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3214, total_rows=22498) [2024-09-26 16:00:50.667128] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.667135] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847408}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.667154] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:1152921504606847408}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9266505480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847408}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650657126, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=10.01ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.667301] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=51] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A6-0-0, dag_ret:0, dag_status:2, start_time:1727337650657094, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665055d0, output_block_cnt_arr:0x7f92665055d8, concurrent_cnt:1, estimate_row_cnt:35098, estimate_occupy_size:18874368, latest_update_ts:1727337650657126, estimated_finish_time:1727337667667309}) [2024-09-26 16:00:50.667309] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264c3c310, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A6-0-0, dag_ret:0, dag_status:2, start_time:1727337650657094, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.667316] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=6] task finish process(ret=0, start_time=1727337650657123, end_time=1727337650667315, runtime=10192, *this={this:0x7f9264c3c310, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A6-0-0, dag_ret:0, dag_status:2, start_time:1727337650657094, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.667342] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337650630011, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.667355] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=12] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:50.667358] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9268515480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.667345] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337650630076, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.667372] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=26] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753327496, storage_schema:{this:0x7f9265ab3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753327496, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.667389] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.667393] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.667399] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265ab3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:444}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753327496, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.667424] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:444}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753327496, storage_schema:{this:0x7f9265ab3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:444}, table_count:1, [{i:0, table_key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8632d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650666636, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=761us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.667472] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=42] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AB-0-0, dag_ret:0, dag_status:2, start_time:1727337650666595, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265ab3490, output_block_cnt_arr:0x7f9265ab3498, concurrent_cnt:1, estimate_row_cnt:4, estimate_occupy_size:6291456, latest_update_ts:1727337650666636, estimated_finish_time:1727337667667475}) [2024-09-26 16:00:50.667480] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264c3dd00, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AB-0-0, dag_ret:0, dag_status:2, start_time:1727337650666595, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.667485] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=5] task finish process(ret=0, start_time=1727337650666633, end_time=1727337650667485, runtime=852, *this={this:0x7f9264c3dd00, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AB-0-0, dag_ret:0, dag_status:2, start_time:1727337650666595, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.667508] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=7] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3223, total_rows=22561) [2024-09-26 16:00:50.667513] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.667520] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847409}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.667538] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:1152921504606847409}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9268515480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847409}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8640e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650662925, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.59ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.667653] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A7-0-0, dag_ret:0, dag_status:2, start_time:1727337650657106, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685155d0, output_block_cnt_arr:0x7f92685155d8, concurrent_cnt:1, estimate_row_cnt:32439, estimate_occupy_size:18874368, latest_update_ts:1727337650662925, estimated_finish_time:1727337667667661}) [2024-09-26 16:00:50.667660] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3c840, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A7-0-0, dag_ret:0, dag_status:2, start_time:1727337650657106, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.667665] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=5] task finish process(ret=0, start_time=1727337650662922, end_time=1727337650667665, runtime=4743, *this={this:0x7f9264c3c840, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A7-0-0, dag_ret:0, dag_status:2, start_time:1727337650657106, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.667685] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=4] schedule one task(task=0x7f9264d763c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.667691] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=4] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.667696] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=2] schedule one task(task=0x7f9264ba8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.667756] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=1152921504606847408, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9272832060, count:5, allocator:0x7f9269ffa7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9272832360, local_datums:0x7f9272832360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:50.667777] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=444, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92659fa060, count:6, allocator:0x7f928643e7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92659fa390, local_datums:0x7f92659fa390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:50.667955] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=23][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:50.667999] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.667998] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=26][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:50.668002] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.668009] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:50.668011] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:50.668013] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f926260c070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f926260c0a8, hex: 0300000000000000, int: 3},idx=2:{len: 12, flag: 0, null: 0, ptr: 0x7f926260c1b0, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f926260c118, hex: 420B000000000000, int: 2882},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f926260c150, hex: 288E681FE7280BE8, int: -1726241059036951000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f926260c188, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:24981, column_count:9, rowkey_column_count:6, has_column_checksum:0, row_count:207, row_store_type:0, opt:0, var_column_count:0, row_offset:15610, original_length:16378, max_merged_trans_version:1726241059036951000, data_length:16378, data_zlength:16378, data_checksum:1574137226, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926f404090, buf_size:16378, data_size:16378, row_count:207, column_count:9, max_merged_trans_version:1726241059036951000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:666643265, row_count_delta:3, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16378}) [2024-09-26 16:00:50.668025] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:50.668028] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:50.668029] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:50.668030] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:50.668032] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:50.668033] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:50.668040] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.668043] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.668050] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:50.668052] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:50.668054] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 12, flag: 0, null: 0, ptr: 0x7f92659fe178, hex: 30323232362D32312D373825, cstr: 02226-21-78%},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f92659fe184, hex: 3230323430393133303435343039, cstr: 20240913045409},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fe0e0, hex: CD7F500300000000, int: 55607245},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fe118, hex: E8EC05CCA0600BE8, int: -1726179788432479000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fe150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-11869, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:238, row_store_type:0, opt:5, var_column_count:0, row_offset:15429, original_length:16321, max_merged_trans_version:1726190643638412000, data_length:16321, data_zlength:5388, data_checksum:152456761, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9262614030, buf_size:5388, data_size:16321, row_count:238, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2114797869, row_count_delta:238, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16321}) [2024-09-26 16:00:50.668066] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=12][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:50.668068] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:50.668069] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:50.668070] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:50.668035] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92659fa390, local_datums:0x7f92659fa390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92659fa390, local_datums:0x7f92659fa390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f928643e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f928643e118, limit_param:null, stmt_allocator:0x7f928643e7d8, allocator:0x7f928643e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:2338610029443177811, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92659fa390, local_datums:0x7f92659fa390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337650630076, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}) [2024-09-26 16:00:50.668094] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=59][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:50.668072] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9272832360, local_datums:0x7f9272832360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92728323e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9272832418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9272832450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9272832360, local_datums:0x7f9272832360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92728323e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9272832418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9272832450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9269ffa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269ffa118, limit_param:null, stmt_allocator:0x7f9269ffa7d8, allocator:0x7f9269ffa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:823131927109750840, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9272832360, local_datums:0x7f9272832360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92728323e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9272832418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9272832450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337650629991, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:50.668123] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=50][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:50.668095] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92659fa390, local_datums:0x7f92659fa390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f928643e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f928643e118, limit_param:null, stmt_allocator:0x7f928643e7d8, allocator:0x7f928643e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:2338610029443177811, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92659fa390, local_datums:0x7f92659fa390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92659fa560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337650630076, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}]) [2024-09-26 16:00:50.668133] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=38][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:50.668135] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=2] partition merge iter row count(i=0, row_count=207, ghost_row_count=0, pkey={tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337650630076, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.668147] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:50.668124] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9272832360, local_datums:0x7f9272832360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92728323e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9272832418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9272832450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9269ffa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269ffa118, limit_param:null, stmt_allocator:0x7f9269ffa7d8, allocator:0x7f9269ffa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:823131927109750840, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9272832360, local_datums:0x7f9272832360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92728323e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9272832418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9272832450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337650629991, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:50.668156] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=32][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:50.668156] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:50.668159] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:50.668158] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=2] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337650629991, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.668160] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337650667725, end_time=1727337650668160, runtime=435, *this={this:0x7f9264ba8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AB-0-0, dag_ret:0, dag_status:2, start_time:1727337650666595, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.668167] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264ba8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AB-0-0, dag_ret:0, dag_status:2, start_time:1727337650666595, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:50.668171] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:50.668174] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=1579, dag_cnt=39, dag_cnts_[dag.get_type()]=39, &dag=0x7f9264c3d940, dag={ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AB-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650666595, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.668181] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:50.668182] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AB-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650666589, task_id:YB42C0A8002D-0006230056D8C6AB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.668184] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:50.668186] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337650667712, end_time=1727337650668186, runtime=474, *this={this:0x7f9264d763c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A6-0-0, dag_ret:0, dag_status:2, start_time:1727337650657094, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.668193] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264d763c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A6-0-0, dag_ret:0, dag_status:2, start_time:1727337650657094, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:50.668214] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=17] dag finished(dag_ret=-4184, runtime=11119, dag_cnt=38, dag_cnts_[dag.get_type()]=38, &dag=0x7f9264c3bf50, dag={ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A6-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650657094, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.668222] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A6-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650657080, task_id:YB42C0A8002D-0006230056D8C6A6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.668246] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650668244, task_id:YB42C0A8002D-0006230056D8C6AC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.668250] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=3] schedule one task(task=0x7f9264c3e230, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.668256] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650668256, task_id:YB42C0A8002D-0006230056D8C6AD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.668259] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=3] schedule one task(task=0x7f9264c3e760, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.668256] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=1152921504606847409, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684aa060, count:5, allocator:0x7f928643e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684aa360, local_datums:0x7f92684aa360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:50.668504] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=30][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:50.668540] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.668543] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.668550] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:50.668552] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:50.668553] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 7, flag: 0, null: 0, ptr: 0x7f92665fe178, hex: 31303033343330, cstr: 1003430},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f92665fe17f, hex: 3230323430393133303435303537, cstr: 20240913045057},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe0e0, hex: 6E48500300000000, int: 55593070},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe118, hex: 58B03BC5A1610BE8, int: -1726178684739801000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-26357, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:250, row_store_type:0, opt:5, var_column_count:0, row_offset:15394, original_length:16334, max_merged_trans_version:1726190643638412000, data_length:16334, data_zlength:5786, data_checksum:2846467531, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92684de030, buf_size:5786, data_size:16334, row_count:250, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:796410710, row_count_delta:250, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16334}) [2024-09-26 16:00:50.668564] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:50.668566] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:50.668567] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:50.668568] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:50.668570] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684aa360, local_datums:0x7f92684aa360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684aa360, local_datums:0x7f92684aa360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f928643e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f928643e118, limit_param:null, stmt_allocator:0x7f928643e7d8, allocator:0x7f928643e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1726241132764173999, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684aa360, local_datums:0x7f92684aa360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337650630011, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:50.668616] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=45][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:50.668617] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684aa360, local_datums:0x7f92684aa360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f928643e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f928643e118, limit_param:null, stmt_allocator:0x7f928643e7d8, allocator:0x7f928643e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1726241132764173999, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684aa360, local_datums:0x7f92684aa360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684aa450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337650630011, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:50.668647] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=30][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:50.668649] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=2] partition merge iter row count(i=0, row_count=250, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337650630011, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.668660] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:50.668669] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:50.668672] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:50.668673] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337650668212, end_time=1727337650668673, runtime=461, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A7-0-0, dag_ret:0, dag_status:2, start_time:1727337650657106, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.668679] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A7-0-0, dag_ret:0, dag_status:2, start_time:1727337650657106, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:50.668686] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=11579, dag_cnt=37, dag_cnts_[dag.get_type()]=37, &dag=0x7f9264c3c480, dag={ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A7-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650657106, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.668694] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A7-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650657102, task_id:YB42C0A8002D-0006230056D8C6A7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.668748] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.668754] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.668757] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.668763] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.668768] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8633ce0, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[816017-1849-0-2305843644870795264-2](offset=1941504,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:331}, data_tablet_id:{id:331}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1433772493, row_count:436}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86343bc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844404006256640-0](offset=1933312,size=1299,type=2,seq=0), type:"MAJOR", tablet_id:{id:331}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726076731692125000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843945518499091-0](offset=1934611,size=872,type=2,seq=0), type:"MINOR", tablet_id:{id:331}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178440167442000, upper_trans_version:1726178440167442000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337650630111, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd32b0, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241465393532000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209503876}, this:0x7f92a3bd32b0, timestamp:1727334209503876, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:64, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277479152983000, v:0}, rec_scn:{val:1726242365923499000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843367845400576-2](offset=1937408,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719356238-0](offset=1937742,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881284074-0](offset=1937898,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447804108-0](offset=1938124,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a77d80, 0x7f92a3bd32b0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.668812] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=44] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.668820] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241465393532000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.668957] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650668954, task_id:YB42C0A8002D-0006230056D8C6AE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.668965] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=7] schedule one task(task=0x7f9264c3ec90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.669035] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.669040] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.669043] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.669051] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.669056] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726176799210367000, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b862fe60, is_inited:true, wash_score:1727337650628844000, hold_ref_cnt:true, ref_cnt:2, version:2, length:497, tablet_addr:[816017-1849-0-2305843637353291776-2](offset=630784,size=585,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:412}, data_tablet_id:{id:412}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726176799210367000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3559841642, row_count:4}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8630518, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844255828574208-0](offset=622592,size=1161,type=2,seq=0), type:"MAJOR", tablet_id:{id:412}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725843390668168000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335631832201-0](offset=623753,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:412}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337650630061, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd6790, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334211402085}, this:0x7f92a3bd6790, timestamp:1727334211402085, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:70, clock:618659840}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277807335692000, v:0}, rec_scn:{val:1726250403931267000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365696606208-2](offset=626688,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170275594572-0](offset=627020,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251879973346-0](offset=627170,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216446493380-0](offset=627396,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a75e40, 0x7f92a3bd6790, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.669100] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=44] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.669120] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.669156] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337650630111, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.669167] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.669180] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.669183] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.669188] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:331}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.669206] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:331}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:331}, table_count:1, [{i:0, table_key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}, merge_scn:{val:1726241465393532000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8633ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650668735, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=451us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.669232] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AD-0-0, dag_ret:0, dag_status:2, start_time:1727337650668259, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3480, output_block_cnt_arr:0x7f92728f3488, concurrent_cnt:1, estimate_row_cnt:45, estimate_occupy_size:0, latest_update_ts:1727337650668735, estimated_finish_time:1727337667669232}) [2024-09-26 16:00:50.669238] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c3e760, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AD-0-0, dag_ret:0, dag_status:2, start_time:1727337650668259, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:50.669243] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=5] task finish process(ret=0, start_time=1727337650668734, end_time=1727337650669243, runtime=509, *this={this:0x7f9264c3e760, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AD-0-0, dag_ret:0, dag_status:2, start_time:1727337650668259, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:50.669301] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=4] schedule one task(task=0x7f926592a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.669362] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=13] multi version scan iterator init succ(param.table_id_=331, range=0x7f92a9ed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b08060, count:5, allocator:0x7f92a33567d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9265b08360, local_datums:0x7f9265b08360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:50.669614] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=16][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:50.669654] ERROR issue_dba_error (ob_log.cpp:1875) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.669658] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.669666] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:50.669670] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=4][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:50.669673] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265ab0070, hex: 0000000000000000, int: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265ab00a8, hex: 3501000000000000, int: 309},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265ab00e0, hex: 3501000000000000, int: 309},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265ab0118, hex: 7032CD318A310BE8, int: -1726231562555674000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265ab0150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-24587, column_count:14, rowkey_column_count:5, has_column_checksum:0, row_count:196, row_store_type:0, opt:0, var_column_count:0, row_offset:15642, original_length:16366, max_merged_trans_version:1726241465393532000, data_length:16366, data_zlength:16366, data_checksum:3184742917, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926fc04090, buf_size:16366, data_size:16366, row_count:196, column_count:14, max_merged_trans_version:1726241465393532000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2306482841, row_count_delta:7, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16366}) [2024-09-26 16:00:50.669684] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=11][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:50.669686] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=1][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:50.669688] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:50.669689] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:50.669691] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:50.669694] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=3][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:50.669696] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9265b08360, local_datums:0x7f9265b08360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b084f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b085a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b085d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:331}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752994696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9265b08360, local_datums:0x7f9265b08360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b084f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b085a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b085d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, store_ctx:{this:0x7f92a3356118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a3356118, limit_param:null, stmt_allocator:0x7f92a33567d8, allocator:0x7f92a33567d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241465393532000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9265b08360, local_datums:0x7f9265b08360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b084f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b085a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b085d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, key:{"BIGINT":0},{"BIGINT":309},{"BIGINT":309}, value_iter:{value:{this=0x7f92830f5dd8 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=15 list_head=0x7f928848eeb8 latest_compact_node=0x7f9283113eb0 max_trans_version={val:1726241465393532000, v:0} max_trans_id=125582299 max_elr_trans_version={val:1726241465393532000, v:0} max_elr_trans_id=125582299 latest_compact_ts=1727334207881700 last_compact_cnt=49 total_trans_node_cnt=141 max_modify_scn={val:1726241465393532000, v:0} min_modify_scn={val:1726179340987317000, v:0}}, version_iter:this=0x7f928848eb78 trans_version={val:1726241465393532000, v:0} scn={val:1726241465393532000, v:0} tx_id={txid:125582299} prev=0x7f928848e850 next=0x7f928848eeb8 modify_count=65 acc_checksum=2418483105 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:42} seq_no=1726241465159151, multi_version_iter:this=0x7f9281a21200 trans_version={val:1726230662165479000, v:0} scn={val:1726230662165479000, v:0} tx_id={txid:125497658} prev=0x7f9281a20ec0 next=0x7f9283111860 modify_count=41 acc_checksum=883883390 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:76} seq_no=1726230662033825, max_committed_trans_version:1726241465393532000, cur_trans_version:{val:1726231562555674000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241465393532000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:196, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337650630111, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:14, cur_pos:0, count:0}}) [2024-09-26 16:00:50.669761] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=64][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:50.669762] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:331}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752994696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9265b08360, local_datums:0x7f9265b08360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b084f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b085a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b085d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, store_ctx:{this:0x7f92a3356118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a3356118, limit_param:null, stmt_allocator:0x7f92a33567d8, allocator:0x7f92a33567d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241465393532000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241465393532000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9265b08360, local_datums:0x7f9265b08360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08370, hex: 0000000000000000, int: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083a8, hex: 3501000000000000, int: 309},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b083e0, hex: 3501000000000000, int: 309},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08418, hex: 7032CD318A310BE8, int: -1726231562555674000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08450, hex: 0000000000000000, int: 0},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112179, hex: 65C7A58EFD130600, int: 1710829601146725},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9283112181, hex: 59D2109BFF210600, int: 1726231562211929},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b084f8, hex: 7103030000000000, int: 197489},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08530, hex: 0000000000000000, int: 0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08568, hex: 6903030000000000, int: 197481},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b085a0, hex: 9B5C030000000000, int: 220315},col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b085d8, hex: 0000000000000000, int: 0},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b08610, hex: 935C030000000000, int: 220307},col_id=13:null,]}, key:{"BIGINT":0},{"BIGINT":309},{"BIGINT":309}, value_iter:{value:{this=0x7f92830f5dd8 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=15 list_head=0x7f928848eeb8 latest_compact_node=0x7f9283113eb0 max_trans_version={val:1726241465393532000, v:0} max_trans_id=125582299 max_elr_trans_version={val:1726241465393532000, v:0} max_elr_trans_id=125582299 latest_compact_ts=1727334207881700 last_compact_cnt=49 total_trans_node_cnt=141 max_modify_scn={val:1726241465393532000, v:0} min_modify_scn={val:1726179340987317000, v:0}}, version_iter:this=0x7f928848eb78 trans_version={val:1726241465393532000, v:0} scn={val:1726241465393532000, v:0} tx_id={txid:125582299} prev=0x7f928848e850 next=0x7f928848eeb8 modify_count=65 acc_checksum=2418483105 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:42} seq_no=1726241465159151, multi_version_iter:this=0x7f9281a21200 trans_version={val:1726230662165479000, v:0} scn={val:1726230662165479000, v:0} tx_id={txid:125497658} prev=0x7f9281a20ec0 next=0x7f9283111860 modify_count=41 acc_checksum=883883390 version=1726212655114383 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:76} seq_no=1726230662033825, max_committed_trans_version:1726241465393532000, cur_trans_version:{val:1726231562555674000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241465393532000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:196, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337650630111, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:14, cur_pos:0, count:0}}]) [2024-09-26 16:00:50.669801] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=38][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:50.669803] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=1] partition merge iter row count(i=0, row_count=196, ghost_row_count=0, pkey={tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, table={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337650630111, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.669815] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:50.669825] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:50.669829] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:50.669831] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337650669316, end_time=1727337650669830, runtime=514, *this={this:0x7f926592a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AD-0-0, dag_ret:0, dag_status:2, start_time:1727337650668259, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:50.669838] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926592a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AD-0-0, dag_ret:0, dag_status:2, start_time:1727337650668259, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:50.669845] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=1586, dag_cnt=36, dag_cnts_[dag.get_type()]=36, &dag=0x7f9264c3e3a0, dag={ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AD-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650668259, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}) [2024-09-26 16:00:50.669854] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AD-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650668256, task_id:YB42C0A8002D-0006230056D8C6AD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.669889] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337650630061, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.669898] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753247280, storage_schema:{this:0x7f9265a11330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828753247280, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.669910] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.669913] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.669918] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265a11330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:412}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753247280, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.669934] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:412}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753247280, storage_schema:{this:0x7f9265a11330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:412}, table_count:1, [{i:0, table_key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650669028, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=888us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.669955] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AA-0-0, dag_ret:0, dag_status:2, start_time:1727337650665204, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265a11470, output_block_cnt_arr:0x7f9265a11478, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337650669028, estimated_finish_time:1727337667669958}) [2024-09-26 16:00:50.669961] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c3d7d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AA-0-0, dag_ret:0, dag_status:2, start_time:1727337650665204, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.669966] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=5] task finish process(ret=0, start_time=1727337650669026, end_time=1727337650669966, runtime=940, *this={this:0x7f9264c3d7d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AA-0-0, dag_ret:0, dag_status:2, start_time:1727337650665204, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.669979] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=5] schedule one task(task=0x7f9264bcc030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.669986] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337650669984, task_id:YB42C0A8002D-0006230056D8C6AF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.669989] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=3] schedule one task(task=0x7f9264c3f1c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.670038] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=412, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:7}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265894060, count:4, allocator:0x7f92636867d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9265894350, local_datums:0x7f9265894350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:50.670059] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=11][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:50.670094] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.670097] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.670105] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:50.670108] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:50.670110] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 23, flag: 0, null: 0, ptr: 0x7f927282e140, hex: 6865617274626561745F736572766963655F65706F6368, cstr: heartbeat_service_epoch},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e0e0, hex: 8808AF9C9C5C0BE8, int: -1726184204453083000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f927282e118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-7627, column_count:7, rowkey_column_count:4, has_column_checksum:0, row_count:1, row_store_type:0, opt:5, var_column_count:0, row_offset:136, original_length:80, max_merged_trans_version:1726184204453083000, data_length:80, data_zlength:80, data_checksum:1549639079, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92b6a04090, buf_size:80, data_size:80, row_count:1, column_count:7, max_merged_trans_version:1726184204453083000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:37835160, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:80}) [2024-09-26 16:00:50.670120] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=10][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:50.670122] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=1][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:00:50.670124] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=1][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:00:50.670126] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=2][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:00:50.670128] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=1][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:00:50.670130] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337650630061, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.670142] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=10][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:50.670149] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:50.670152] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:50.670154] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337650670003, end_time=1727337650670154, runtime=151, *this={this:0x7f9264bcc030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AA-0-0, dag_ret:0, dag_status:2, start_time:1727337650665204, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.670159] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=5][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264bcc030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AA-0-0, dag_ret:0, dag_status:2, start_time:1727337650665204, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:50.670166] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=4962, dag_cnt=35, dag_cnts_[dag.get_type()]=35, &dag=0x7f9264c3d410, dag={ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AA-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650665204, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.670173] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AA-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650665197, task_id:YB42C0A8002D-0006230056D8C6AA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.670191] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337650670190, task_id:YB42C0A8002D-0006230056D8C6B0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.670194] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=3] schedule one task(task=0x7f9264c3f6f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.670722] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.670729] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.670732] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.670738] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.670743] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8627b60, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:523, tablet_addr:[816017-1849-0-2305843665271992320-2](offset=2043904,size=611,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:458}, data_tablet_id:{id:458}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2897668175, row_count:77572}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86282a8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844621975949312-0](offset=2035712,size=1502,type=2,seq=0), type:"MAJOR", tablet_id:{id:458}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389055265000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844927992370654-0](offset=2037214,size=1787,type=2,seq=0), type:"MINOR", tablet_id:{id:458}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178190631204000, upper_trans_version:1726178190631204000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337650630128, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd1b40, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209496526}, this:0x7f92a3bd1b40, timestamp:1727334209496526, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:61, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274225933800000, v:0}, rec_scn:{val:1726241806895572002, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771761152-2](offset=2039808,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046811469-0](offset=2040141,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881386491-0](offset=2040315,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447906525-0](offset=2040541,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7a490, 0x7f92a3bd1b40, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.670790] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=46] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.670798] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.670835] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.670838] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.670841] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.670845] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.670848] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8631da0, is_inited:true, wash_score:1727337650628844000, hold_ref_cnt:true, ref_cnt:2, version:2, length:515, tablet_addr:[816017-1849-0-2305843656681984000-2](offset=1970176,size=603,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:309}, data_tablet_id:{id:309}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:810117565, row_count:8}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86324c4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844526412836864-0](offset=1945600,size=1413,type=2,seq=0), type:"MAJOR", tablet_id:{id:309}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077442448649001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843990615668101-0](offset=1947013,size=914,type=2,seq=0), type:"MINOR", tablet_id:{id:309}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178572306341000, upper_trans_version:1726178572306341000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337650630094, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcdcc0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241316899724000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405581}, this:0x7f92a3bcdcc0, timestamp:1727334209405581, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:51, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277651138194000, v:0}, rec_scn:{val:1726241616921722000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697937408-2](offset=1957888,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843189604278604-0](offset=1958220,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881304564-0](offset=1958388,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447824598-0](offset=1958614,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a76de0, 0x7f92a3bcdcc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.670885] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=37] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.670891] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241316899724000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.671319] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337650630094, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.671335] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752918752, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828752918752, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.671351] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.671355] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.671361] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:309}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752918752, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.671382] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:309}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752918752, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:309}, table_count:1, [{i:0, table_key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}, merge_scn:{val:1726241316899724000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8631da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650670831, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=528us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.671406] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AC-0-0, dag_ret:0, dag_status:2, start_time:1727337650668249, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3490, output_block_cnt_arr:0x7f92728f3498, concurrent_cnt:1, estimate_row_cnt:6, estimate_occupy_size:6291456, latest_update_ts:1727337650670831, estimated_finish_time:1727337667671409}) [2024-09-26 16:00:50.671413] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3e230, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AC-0-0, dag_ret:0, dag_status:2, start_time:1727337650668249, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:50.671419] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=6] task finish process(ret=0, start_time=1727337650670829, end_time=1727337650671419, runtime=590, *this={this:0x7f9264c3e230, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AC-0-0, dag_ret:0, dag_status:2, start_time:1727337650668249, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:50.671450] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=3] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.671710] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=309, range=0x7f92ab056200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:11}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b0060, count:7, allocator:0x7f92636867d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92684b03b0, local_datums:0x7f92684b03b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:50.671887] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=20][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:50.671934] ERROR issue_dba_error (ob_log.cpp:1875) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.671939] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.671947] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:50.671949] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:50.671952] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=2][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.671968] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=0] partition merge iter row count(i=0, row_count=156, ghost_row_count=0, pkey={tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, table={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337650630094, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.671990] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650671660, end_time=1727337650671990, runtime=330, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AC-0-0, dag_ret:0, dag_status:2, start_time:1727337650668249, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:50.672001] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3752, dag_cnt=34, dag_cnts_[dag.get_type()]=34, &dag=0x7f9264c3de70, dag={ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AC-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650668249, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}) [2024-09-26 16:00:50.672009] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AC-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650668244, task_id:YB42C0A8002D-0006230056D8C6AC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.672044] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.672047] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.672049] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.672054] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.672058] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.672065] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.672085] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.672088] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.672089] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.672095] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.672098] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b863cf80, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:522, tablet_addr:[816017-1849-0-2305843664198197248-2](offset=1990656,size=610,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:487}, data_tablet_id:{id:487}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1549633270, row_count:11693}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863d6c8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844647745695744-0](offset=1978368,size=1526,type=2,seq=0), type:"MAJOR", tablet_id:{id:487}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389060348000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843977730799094-0](offset=1979894,size=902,type=2,seq=0), type:"MINOR", tablet_id:{id:487}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178190844721000, upper_trans_version:1726178190844721000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337650630145, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd4850, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209506557}, this:0x7f92a3bd4850, timestamp:1727334209506557, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:66, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274231198751000, v:0}, rec_scn:{val:1726241813664082000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843367845449728-2](offset=1986560,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046758222-0](offset=1986894,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881333244-0](offset=1987068,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447853278-0](offset=1987294,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a79cc0, 0x7f92a3bd4850, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.672165] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=67] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.672171] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.672245] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337650630128, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.672256] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753357664, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753357664, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.672317] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:50.672321] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.672326] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:458}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753357664, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.672346] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:458}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753357664, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:458}, table_count:1, [{i:0, table_key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8627b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650670714, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.61ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.672475] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=22] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:50.672644] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AE-0-0, dag_ret:0, dag_status:2, start_time:1727337650668965, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665054a0, output_block_cnt_arr:0x7f92665054a8, concurrent_cnt:1, estimate_row_cnt:14598, estimate_occupy_size:4194304, latest_update_ts:1727337650670714, estimated_finish_time:1727337667672645}) [2024-09-26 16:00:50.672677] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=32] succeed to init merge ctx(task={this:0x7f9264c3ec90, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AE-0-0, dag_ret:0, dag_status:2, start_time:1727337650668965, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.672684] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=7] task finish process(ret=0, start_time=1727337650670712, end_time=1727337650672684, runtime=1972, *this={this:0x7f9264c3ec90, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AE-0-0, dag_ret:0, dag_status:2, start_time:1727337650668965, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.672734] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=4] schedule one task(task=0x7f9268486030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.672743] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337650672741, task_id:YB42C0A8002D-0006230056D8C6B1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.672748] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=4] schedule one task(task=0x7f9264c3fc20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.672765] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.672769] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.672771] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.672775] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.672779] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8624c80, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:541, tablet_addr:[816017-1849-0-2305843684599267328-2](offset=1966080,size=629,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:492}, data_tablet_id:{id:492}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2528622955, row_count:340306}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8625434, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305850166778642432-0](offset=1949696,size=6666,type=2,seq=0), type:"MAJOR", tablet_id:{id:492}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389176297000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844149529467402-0](offset=1956362,size=1062,type=2,seq=0), type:"MINOR", tablet_id:{id:492}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178192725955000, upper_trans_version:1726178192725955000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337650630177, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd2310, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209496637}, this:0x7f92a3bd2310, timestamp:1727334209496637, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:62, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274246934057000, v:0}, rec_scn:{val:1726241820481609000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697941504-2](offset=1961984,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843215374086476-0](offset=1962316,size=192,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881308684-0](offset=1962508,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447828718-0](offset=1962734,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a78550, 0x7f92a3bd2310, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.672818] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=38] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.672823] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.673296] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337650630161, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.673312] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753332736, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=36, max_schema_version_in_memtable=1710828753332736, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.673452] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:50.673457] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.673464] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:455}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:10, rowkey_column_count:9, schema_rowkey_col_cnt:7, full_stored_col_cnt:38, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753332736, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=26 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.673488] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=18] succeed to build merge ctx(tablet_id={id:455}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753332736, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:455}, table_count:1, [{i:0, table_key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650672038, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.42ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.673666] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=21] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:50.673825] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B0-0-0, dag_ret:0, dag_status:2, start_time:1727337650670194, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685154b0, output_block_cnt_arr:0x7f92685154b8, concurrent_cnt:1, estimate_row_cnt:3824, estimate_occupy_size:4194304, latest_update_ts:1727337650672038, estimated_finish_time:1727337667673827}) [2024-09-26 16:00:50.673835] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=10] succeed to init merge ctx(task={this:0x7f9264c3f6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B0-0-0, dag_ret:0, dag_status:2, start_time:1727337650670194, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.673864] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=28] task finish process(ret=0, start_time=1727337650672036, end_time=1727337650673863, runtime=1827, *this={this:0x7f9264c3f6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B0-0-0, dag_ret:0, dag_status:2, start_time:1727337650670194, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.673901] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337650630177, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.673913] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753437040, storage_schema:{this:0x7f92664f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=57, max_schema_version_in_memtable=1710828753437040, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.673974] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1362, total_rows=1362) [2024-09-26 16:00:50.673977] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.673983] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92664f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:492}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:12, rowkey_column_count:11, schema_rowkey_col_cnt:9, full_stored_col_cnt:59, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753437040, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=28 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.674008] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=20] succeed to build merge ctx(tablet_id={id:492}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753437040, storage_schema:{this:0x7f92664f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:492}, table_count:1, [{i:0, table_key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8624c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650672760, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.22ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.674079] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B1-0-0, dag_ret:0, dag_status:2, start_time:1727337650672747, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92664f14c0, output_block_cnt_arr:0x7f92664f14c8, concurrent_cnt:1, estimate_row_cnt:1362, estimate_occupy_size:2097152, latest_update_ts:1727337650672760, estimated_finish_time:1727337667674079}) [2024-09-26 16:00:50.674085] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c3fc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B1-0-0, dag_ret:0, dag_status:2, start_time:1727337650672747, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.674090] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=5] task finish process(ret=0, start_time=1727337650672759, end_time=1727337650674090, runtime=1331, *this={this:0x7f9264c3fc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B1-0-0, dag_ret:0, dag_status:2, start_time:1727337650672747, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.674123] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337650630145, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.674133] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753417144, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828753417144, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.674201] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:50.674206] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.674229] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337650328055]}) [2024-09-26 16:00:50.674220] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:487}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753417144, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.674249] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=24] succeed to build merge ctx(tablet_id={id:487}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753417144, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:487}, table_count:1, [{i:0, table_key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650672082, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.13ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.674333] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=17] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:50.674431] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AF-0-0, dag_ret:0, dag_status:2, start_time:1727337650669989, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab4a0, output_block_cnt_arr:0x7f92684ab4a8, concurrent_cnt:1, estimate_row_cnt:2199, estimate_occupy_size:0, latest_update_ts:1727337650672082, estimated_finish_time:1727337667674431}) [2024-09-26 16:00:50.674438] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3f1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AF-0-0, dag_ret:0, dag_status:2, start_time:1727337650669989, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.674444] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=5] task finish process(ret=0, start_time=1727337650672081, end_time=1727337650674443, runtime=2362, *this={this:0x7f9264c3f1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AF-0-0, dag_ret:0, dag_status:2, start_time:1727337650669989, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.674497] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=2] schedule one task(task=0x7f9262cbe030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.674506] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=5] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.674510] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=3] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.674570] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=492, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:57, schema_rowkey_cnt:9, rowkey_cnt:11, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:9, column_cnt:59}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265898060, count:11, allocator:0x7f92bc8f87d8, capacity:11, init_cnt:11}, datum_utils:{is_oracle_mode:false, rowkey_cnt:11, col_cnt:11, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:59, datum_buffer:{capacity:59, datums:0x7f9264c88050, local_datums:0x7f9265898440}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,col_id=43:NOP,col_id=44:NOP,col_id=45:NOP,col_id=46:NOP,col_id=47:NOP,col_id=48:NOP,col_id=49:NOP,col_id=50:NOP,col_id=51:NOP,col_id=52:NOP,col_id=53:NOP,col_id=54:NOP,col_id=55:NOP,col_id=56:NOP,col_id=57:NOP,col_id=58:NOP,]}) [2024-09-26 16:00:50.674783] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=20][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.674828] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.674834] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.674845] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=0] partition merge iter row count(i=0, row_count=61, ghost_row_count=0, pkey={tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337650630177, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.674870] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650674522, end_time=1727337650674870, runtime=348, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B1-0-0, dag_ret:0, dag_status:2, start_time:1727337650672747, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.674880] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2133, dag_cnt=33, dag_cnts_[dag.get_type()]=33, &dag=0x7f9264c3f860, dag={ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B1-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650672747, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.674888] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B1-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650672741, task_id:YB42C0A8002D-0006230056D8C6B1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.674910] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650674908, task_id:YB42C0A8002D-0006230056D8C6B2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.674914] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=3] schedule one task(task=0x7f9264c40150, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.674946] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.674951] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.674953] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.674957] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.674961] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b863fe60, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:511, tablet_addr:[816017-1849-0-2305843652386852864-2](offset=1806336,size=599,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:501}, data_tablet_id:{id:501}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1792252290, row_count:320564}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8640560, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305866008765333504-0](offset=1769472,size=21420,type=2,seq=0), type:"MAJOR", tablet_id:{id:501}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389518729000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844002426672044-0](offset=1790892,size=925,type=2,seq=0), type:"MINOR", tablet_id:{id:501}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178194873791000, upper_trans_version:1726178194873791000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337650630195, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd03d0, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495744}, this:0x7f92a3bd03d0, timestamp:1727334209495744, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:58, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274254042757000, v:0}, rec_scn:{val:1726241824983317000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771515392-2](offset=1794048,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161663821-0](offset=1794381,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881140719-0](offset=1794543,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843215373918929-0](offset=1794769,size=192,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a78d20, 0x7f92a3bd03d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.675002] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=41] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.675009] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.675438] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=487, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:11}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c72060, count:8, allocator:0x7f92865fe7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9262c723e0, local_datums:0x7f9262c723e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:50.675692] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.675735] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=8][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.675739] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.675751] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=0] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337650630145, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.675785] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650675367, end_time=1727337650675785, runtime=418, *this={this:0x7f9262cbe030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AF-0-0, dag_ret:0, dag_status:2, start_time:1727337650669989, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.675795] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5806, dag_cnt=32, dag_cnts_[dag.get_type()]=32, &dag=0x7f9264c3ee00, dag={ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AF-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650669989, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.675804] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AF-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337650669984, task_id:YB42C0A8002D-0006230056D8C6AF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.675862] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=455, range=0x7f92aa952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:36, schema_rowkey_cnt:7, rowkey_cnt:9, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:7, column_cnt:38}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c6e060, count:9, allocator:0x7f9269ffa7d8, capacity:9, init_cnt:9}, datum_utils:{is_oracle_mode:false, rowkey_cnt:9, col_cnt:9, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:38, datum_buffer:{capacity:38, datums:0x7f9262c6f340, local_datums:0x7f9262c6e3f0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,]}) [2024-09-26 16:00:50.676114] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.676147] ERROR issue_dba_error (ob_log.cpp:1875) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.676150] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.676159] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=0] partition merge iter row count(i=0, row_count=140, ghost_row_count=0, pkey={tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337650630161, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.676178] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650675830, end_time=1727337650676178, runtime=348, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B0-0-0, dag_ret:0, dag_status:2, start_time:1727337650670194, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.676187] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5993, dag_cnt=31, dag_cnts_[dag.get_type()]=31, &dag=0x7f9264c3f330, dag={ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B0-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650670194, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.676194] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B0-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650670190, task_id:YB42C0A8002D-0006230056D8C6B0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.676275] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=458, range=0x7f92aa352200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b08060, count:8, allocator:0x7f92ba7f27d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9265b083e0, local_datums:0x7f9265b083e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:50.676513] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.676545] ERROR issue_dba_error (ob_log.cpp:1875) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.676548] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.676564] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=0] partition merge iter row count(i=0, row_count=247, ghost_row_count=0, pkey={tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337650630128, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.676585] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650676236, end_time=1727337650676585, runtime=349, *this={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AE-0-0, dag_ret:0, dag_status:2, start_time:1727337650668965, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.676594] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=7629, dag_cnt=30, dag_cnts_[dag.get_type()]=30, &dag=0x7f9264c3e8d0, dag={ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6AE-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650668965, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.676601] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6AE-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650668954, task_id:YB42C0A8002D-0006230056D8C6AE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.676636] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337650630195, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.676648] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753441792, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828753441792, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.676704] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1295, total_rows=1295) [2024-09-26 16:00:50.676708] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.676713] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:501}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753441792, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.676739] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337650676737, task_id:YB42C0A8002D-0006230056D8C6B3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.676743] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=4] schedule one task(task=0x7f9264c40680, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.676732] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:501}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753441792, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:501}, table_count:1, [{i:0, table_key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650674940, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.77ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.676748] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337650676748, task_id:YB42C0A8002D-0006230056D8C6B4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.676751] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=3] schedule one task(task=0x7f9264c40bb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.676755] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650676754, task_id:YB42C0A8002D-0006230056D8C6B5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.676757] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=1] schedule one task(task=0x7f9264c410e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.676775] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.676779] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.676781] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.676785] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.676788] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.676795] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.676808] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B2-0-0, dag_ret:0, dag_status:2, start_time:1727337650674913, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685e7490, output_block_cnt_arr:0x7f92685e7498, concurrent_cnt:1, estimate_row_cnt:1295, estimate_occupy_size:0, latest_update_ts:1727337650674940, estimated_finish_time:1727337667676808}) [2024-09-26 16:00:50.676815] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c40150, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B2-0-0, dag_ret:0, dag_status:2, start_time:1727337650674913, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.676820] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=5] task finish process(ret=0, start_time=1727337650674937, end_time=1727337650676820, runtime=1883, *this={this:0x7f9264c40150, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B2-0-0, dag_ret:0, dag_status:2, start_time:1727337650674913, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.676828] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.676839] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=10][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.676840] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726077706339704027) [2024-09-26 16:00:50.676844] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.676846] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726077706339704027, tablet={this:0x7f92b864a0a0, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[720905-1289-0-2305843650239553536-1](offset=1990656,size=597,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:342}, data_tablet_id:{id:342}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725843677180998000, v:0}, clog_checkpoint_scn:{val:1725843677180998000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726077706339704027, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2338495768, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864a7a0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[720905-1289-0-2305844556477644800-0](offset=1982464,size=1441,type=2,seq=0), type:"MAJOR", tablet_id:{id:342}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725843393342764001}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337650630227, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[720905-1289-0-2305843251881332736-1](offset=1986560,size=226,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[720905-1289-0-2305843183161856226-0](offset=1986786,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[720905-1289-0-2305843251881333124-0](offset=1986948,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[720905-1289-0-2305843216447853158-0](offset=1987174,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7cba0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:50.676880] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=33] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.676884] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.676901] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.676904] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.676906] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.676909] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.676912] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b862a0a0, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:490, tablet_addr:[816017-1849-0-2305843629838340096-2](offset=1871872,size=578,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:472}, data_tablet_id:{id:472}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:19976489, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862a734, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844179594129408-0](offset=1847296,size=1090,type=2,seq=0), type:"MAJOR", tablet_id:{id:472}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389523010000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843910084932674-0](offset=1848386,size=839,type=2,seq=0), type:"MINOR", tablet_id:{id:472}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178194887823000, upper_trans_version:1726178194887823000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337650630211, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd4080, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209505593}, this:0x7f92a3bd4080, timestamp:1727334209505593, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:65, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274254072414000, v:0}, rec_scn:{val:1726241829658678000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697839104-2](offset=1859584,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834376524-0](offset=1859916,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881206236-0](offset=1860060,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447726270-0](offset=1860286,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7ac60, 0x7f92a3bd4080, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.676942] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=30] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.676946] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.676995] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=1] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.677043] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=501, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265896060, count:6, allocator:0x7f92bc8fa7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265896390, local_datums:0x7f9265896390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:50.677140] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.677188] ERROR issue_dba_error (ob_log.cpp:1875) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.677191] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.677201] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=0] partition merge iter row count(i=0, row_count=56, ghost_row_count=0, pkey={tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337650630195, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.677221] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650677008, end_time=1727337650677220, runtime=212, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B2-0-0, dag_ret:0, dag_status:2, start_time:1727337650674913, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.677231] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2318, dag_cnt=29, dag_cnts_[dag.get_type()]=29, &dag=0x7f9264c3fd90, dag={ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B2-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650674913, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.677239] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B2-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650674908, task_id:YB42C0A8002D-0006230056D8C6B2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.677274] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337650630243, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.677285] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.677297] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.677302] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.677307] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:140}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.677324] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:140}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:140}, table_count:1, [{i:0, table_key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8622d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650676769, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=537us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.677347] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B5-0-0, dag_ret:0, dag_status:2, start_time:1727337650676756, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab460, output_block_cnt_arr:0x7f92684ab468, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337650676769, estimated_finish_time:1727337667677347}) [2024-09-26 16:00:50.677353] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c410e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B5-0-0, dag_ret:0, dag_status:2, start_time:1727337650676756, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.677358] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=5] task finish process(ret=0, start_time=1727337650676767, end_time=1727337650677358, runtime=591, *this={this:0x7f9264c410e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B5-0-0, dag_ret:0, dag_status:2, start_time:1727337650676756, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.677374] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337650630211, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.677382] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753360568, storage_schema:{this:0x7f9265ab5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753360568, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.677389] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=5] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.677392] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.677396] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265ab5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:472}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753360568, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.677415] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:472}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753360568, storage_schema:{this:0x7f9265ab5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:472}, table_count:1, [{i:0, table_key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650676898, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=497us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.677430] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=12] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B3-0-0, dag_ret:0, dag_status:2, start_time:1727337650676743, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265ab5460, output_block_cnt_arr:0x7f9265ab5468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337650676898, estimated_finish_time:1727337667677430}) [2024-09-26 16:00:50.677437] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c40680, type:15, status:2, dag:{ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B3-0-0, dag_ret:0, dag_status:2, start_time:1727337650676743, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.677442] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=5] task finish process(ret=0, start_time=1727337650676897, end_time=1727337650677442, runtime=545, *this={this:0x7f9264c40680, type:15, status:2, dag:{ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B3-0-0, dag_ret:0, dag_status:2, start_time:1727337650676743, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.677474] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337650630227, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.677484] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753001832, storage_schema:{this:0x7f9272839330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=21, max_schema_version_in_memtable=1710828753001832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.677493] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.677496] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.677500] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9272839330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:342}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:23, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753001832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.677516] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:342}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753001832, storage_schema:{this:0x7f9272839330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:342}, table_count:1, [{i:0, table_key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650676824, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=675us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.677562] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B4-0-0, dag_ret:0, dag_status:2, start_time:1727337650676751, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9272839490, output_block_cnt_arr:0x7f9272839498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337650676824, estimated_finish_time:1727337667677561}) [2024-09-26 16:00:50.677571] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264c40bb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B4-0-0, dag_ret:0, dag_status:2, start_time:1727337650676751, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.677577] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=5] task finish process(ret=0, start_time=1727337650676823, end_time=1727337650677576, runtime=753, *this={this:0x7f9264c40bb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B4-0-0, dag_ret:0, dag_status:2, start_time:1727337650676751, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.677592] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=3] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.677599] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=4] schedule one task(task=0x7f9264c49260, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.677605] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=5] schedule one task(task=0x7f9287cf83c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.677610] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650677608, task_id:YB42C0A8002D-0006230056D8C6B6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.677613] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=2] schedule one task(task=0x7f9264c41610, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.677648] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=472, range=0x7f92aaf52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685ea060, count:3, allocator:0x7f92c19467d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f92685ea320, local_datums:0x7f92685ea320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:50.677664] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=140, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dd2060, count:3, allocator:0x7f92636867d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287dd2320, local_datums:0x7f9287dd2320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:50.677686] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.677713] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.677725] ERROR issue_dba_error (ob_log.cpp:1875) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.677728] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.677738] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=0] partition merge iter row count(i=0, row_count=17, ghost_row_count=0, pkey={tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337650630211, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.677747] ERROR issue_dba_error (ob_log.cpp:1875) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.677749] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.677758] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650677616, end_time=1727337650677757, runtime=141, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B3-0-0, dag_ret:0, dag_status:2, start_time:1727337650676743, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.677758] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337650630243, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.677767] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1024, dag_cnt=28, dag_cnts_[dag.get_type()]=28, &dag=0x7f9264c402c0, dag={ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B3-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650676743, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.677775] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B3-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650676737, task_id:YB42C0A8002D-0006230056D8C6B3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.677778] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650677624, end_time=1727337650677778, runtime=154, *this={this:0x7f9287cf83c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B5-0-0, dag_ret:0, dag_status:2, start_time:1727337650676756, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.677841] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=342, range=0x7f92aac56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:21, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:23}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b08060, count:6, allocator:0x7f92636867d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:23, datum_buffer:{capacity:32, datums:0x7f9265b08390, local_datums:0x7f9265b08390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,]}) [2024-09-26 16:00:50.677871] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.677902] ERROR issue_dba_error (ob_log.cpp:1875) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.677905] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.677913] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337650630227, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.677929] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650677804, end_time=1727337650677929, runtime=125, *this={this:0x7f9264c49260, type:1, status:2, dag:{ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B4-0-0, dag_ret:0, dag_status:2, start_time:1727337650676751, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.677938] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=1] dag finished(dag_ret=-4184, runtime=1187, dag_cnt=27, dag_cnts_[dag.get_type()]=27, &dag=0x7f9264c407f0, dag={ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B4-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650676751, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.677946] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B4-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650676748, task_id:YB42C0A8002D-0006230056D8C6B4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.677964] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=1] dag finished(dag_ret=-4184, runtime=1208, dag_cnt=26, dag_cnts_[dag.get_type()]=26, &dag=0x7f9264c40d20, dag={ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B5-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650676756, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.677970] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B5-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650676754, task_id:YB42C0A8002D-0006230056D8C6B5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.677989] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337650677987, task_id:YB42C0A8002D-0006230056D8C6B7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.677992] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=3] schedule one task(task=0x7f9264c41b40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.678000] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337650677999, task_id:YB42C0A8002D-0006230056D8C6B8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.678003] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=2] schedule one task(task=0x7f9264c42070, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.678010] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337650678009, task_id:YB42C0A8002D-0006230056D8C6B9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.678012] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=2] schedule one task(task=0x7f9264c425a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.678486] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.678493] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.678496] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.678501] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.678505] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.678521] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=11] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.678546] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.678549] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.678551] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:50.678554] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.678557] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.678562] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.678577] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.678580] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.678581] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:50.678586] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.678589] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864df20, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:503, tablet_addr:[814042-5586-0-2305843643796283392-2](offset=1171456,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:377}, data_tablet_id:{id:377}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4164021307, row_count:17}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864e5fc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844760487772160-0](offset=1163264,size=1631,type=2,seq=0), type:"MAJOR", tablet_id:{id:377}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563080818000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843989541144159-0](offset=1164895,size=913,type=2,seq=0), type:"MINI", tablet_id:{id:377}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557779816000, upper_trans_version:1726121557779816000}, {i:1, addr:[814042-5586-0-2305843969140050416-0](offset=1165808,size=894,type=2,seq=0), type:"MINI", tablet_id:{id:377}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986451466000, upper_trans_version:1726149986451466000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337650630308, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdb5b0, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220773832}, this:0x7f92a3bdb5b0, timestamp:1727334220773832, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:80, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293334630000, v:0}, rec_scn:{val:1726264800181268000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513780224-2](offset=1167360,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843176718586294-0](offset=1167798,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880514130-0](offset=1167954,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447034164-0](offset=1168180,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7e310, 0x7f92a3bdb5b0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.678633] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=44] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.678637] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.679408] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.679416] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.679419] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:50.679423] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.679427] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8641da0, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:487, tablet_addr:[816017-1849-0-2305843626617249792-2](offset=2007040,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101009}, data_tablet_id:{id:140}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726073917016589000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3251381279, row_count:150}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8642458, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844188184215552-0](offset=1998848,size=1098,type=2,seq=0), type:"MAJOR", tablet_id:{id:101009}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726070403667750001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843957329724490-0](offset=1999946,size=883,type=2,seq=0), type:"MINOR", tablet_id:{id:101009}, scn_range:{start_scn:{val:1726073917016589000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726167395874925000, upper_trans_version:1726167395874925000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337650630258, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd0ba0, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495816}, this:0x7f92a3bd0ba0, timestamp:1727334209495816, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:59, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726256203821911000, v:0}, rec_scn:{val:1726241803398655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697982464-2](offset=2002944,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276970828-0](offset=2003276,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881349602-0](offset=2003426,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7bc00, 0x7f92a3bd0ba0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.679468] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=41] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.679475] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.679701] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337650630291, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.679715] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752949376, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=20, max_schema_version_in_memtable=1710828752949376, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.679738] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.679743] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.679750] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:323}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:22, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752949376, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.679795] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=39] succeed to build merge ctx(tablet_id={id:323}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752949376, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:323}, table_count:1, [{i:0, table_key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650678543, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.20ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.679832] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B8-0-0, dag_ret:0, dag_status:2, start_time:1727337650678002, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505480, output_block_cnt_arr:0x7f9266505488, concurrent_cnt:1, estimate_row_cnt:205, estimate_occupy_size:0, latest_update_ts:1727337650678543, estimated_finish_time:1727337667679832}) [2024-09-26 16:00:50.679840] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c42070, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B8-0-0, dag_ret:0, dag_status:2, start_time:1727337650678002, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.679846] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=5] task finish process(ret=0, start_time=1727337650678541, end_time=1727337650679845, runtime=1304, *this={this:0x7f9264c42070, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B8-0-0, dag_ret:0, dag_status:2, start_time:1727337650678002, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.679867] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337650630274, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.679877] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.679885] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.679888] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.679893] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101010}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:50.679907] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:101010}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101010}, table_count:1, [{i:0, table_key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8642d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650678479, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.41ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.679930] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B7-0-0, dag_ret:0, dag_status:2, start_time:1727337650677992, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3470, output_block_cnt_arr:0x7f92728f3478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337650678479, estimated_finish_time:1727337667679930}) [2024-09-26 16:00:50.679937] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c41b40, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B7-0-0, dag_ret:0, dag_status:2, start_time:1727337650677992, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.679943] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=6] task finish process(ret=0, start_time=1727337650678477, end_time=1727337650679943, runtime=1466, *this={this:0x7f9264c41b40, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B7-0-0, dag_ret:0, dag_status:2, start_time:1727337650677992, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.679966] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337650630258, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.679977] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9265ab5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.679989] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.679993] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.679998] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265ab5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101009}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:50.680016] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101009}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9265ab5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101009}, table_count:1, [{i:0, table_key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8641da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650679402, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=595us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.680036] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B6-0-0, dag_ret:0, dag_status:2, start_time:1727337650677613, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265ab5470, output_block_cnt_arr:0x7f9265ab5478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337650679402, estimated_finish_time:1727337667680036}) [2024-09-26 16:00:50.680042] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c41610, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B6-0-0, dag_ret:0, dag_status:2, start_time:1727337650677613, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.680047] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=5] task finish process(ret=0, start_time=1727337650679400, end_time=1727337650680047, runtime=647, *this={this:0x7f9264c41610, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B6-0-0, dag_ret:0, dag_status:2, start_time:1727337650677613, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.680061] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=2] schedule one task(task=0x7f9287cf83c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.680068] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=5] schedule one task(task=0x7f9264d763c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.680073] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=4] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.680115] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101010, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685ea060, count:4, allocator:0x7f9269ffa7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92685ea350, local_datums:0x7f92685ea350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:50.680153] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=13][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:50.680194] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.680197] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.680225] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=28][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:50.680229] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=4][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:50.680231] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 16, flag: 0, null: 0, ptr: 0x7f92665fe140, hex: 73746F705F726F6F7473657276696365, cstr: stop_rootservice},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe150, hex: 1E738793F4210600, int: 1726184191128350},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe0e0, hex: C0DCFDA09C5C0BE8, int: -1726184204380808000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19649, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:20, row_store_type:0, opt:5, var_column_count:0, row_offset:1259, original_length:1279, max_merged_trans_version:1726239607937620000, data_length:1279, data_zlength:1279, data_checksum:2320751487, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f926f404090, buf_size:1279, data_size:1279, row_count:20, column_count:4, max_merged_trans_version:1726239607937620000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1439706941, row_count_delta:20, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:1279}) [2024-09-26 16:00:50.680245] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=14][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:50.680247] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=2][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:00:50.680250] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=2][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:00:50.680251] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=1][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:00:50.680254] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=2][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:00:50.680257] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=2] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337650630274, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.680271] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=10][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:50.680280] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:50.680284] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:50.680286] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337650680084, end_time=1727337650680286, runtime=202, *this={this:0x7f9264d763c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B7-0-0, dag_ret:0, dag_status:2, start_time:1727337650677992, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.680292] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264d763c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B7-0-0, dag_ret:0, dag_status:2, start_time:1727337650677992, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:50.680300] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=2308, dag_cnt=25, dag_cnts_[dag.get_type()]=25, &dag=0x7f9264c41780, dag={ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B7-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650677992, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.680311] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B7-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650677987, task_id:YB42C0A8002D-0006230056D8C6B7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.680359] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650680357, task_id:YB42C0A8002D-0006230056D8C6BA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.680364] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=4] schedule one task(task=0x7f9264c42ad0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.680403] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.680408] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.680410] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:50.680415] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.680418] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.680426] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.680532] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=323, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:20, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:22}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dd4060, count:5, allocator:0x7f928643e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f9287dd4360, local_datums:0x7f9287dd4360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,]}) [2024-09-26 16:00:50.680768] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.680811] ERROR issue_dba_error (ob_log.cpp:1875) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=8][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.680814] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.680826] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=0] partition merge iter row count(i=0, row_count=171, ghost_row_count=0, pkey={tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337650630291, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.680848] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650680491, end_time=1727337650680848, runtime=357, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B8-0-0, dag_ret:0, dag_status:2, start_time:1727337650678002, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.680857] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2855, dag_cnt=24, dag_cnts_[dag.get_type()]=24, &dag=0x7f9264c41cb0, dag={ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B8-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650678002, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.680865] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B8-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650677999, task_id:YB42C0A8002D-0006230056D8C6B8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.680895] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337650680893, task_id:YB42C0A8002D-0006230056D8C6BB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.680898] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=3] schedule one task(task=0x7f9264c43000, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.680913] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337650630325, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.680940] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=26] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=16, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.680952] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.680955] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.680954] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.680961] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.680964] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:50.680969] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.680960] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:297}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:18, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.680972] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.680980] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.680979] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:297}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:297}, table_count:1, [{i:0, table_key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650680397, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=561us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.681008] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=24] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BA-0-0, dag_ret:0, dag_status:2, start_time:1727337650680363, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db7490, output_block_cnt_arr:0x7f9287db7498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337650680397, estimated_finish_time:1727337667681008}) [2024-09-26 16:00:50.681015] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c42ad0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BA-0-0, dag_ret:0, dag_status:2, start_time:1727337650680363, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.681020] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=5] task finish process(ret=0, start_time=1727337650680395, end_time=1727337650681020, runtime=625, *this={this:0x7f9264c42ad0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BA-0-0, dag_ret:0, dag_status:2, start_time:1727337650680363, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.681066] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=4] schedule one task(task=0x7f9264bd6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.681120] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=297, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:16, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:18}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dd8060, count:6, allocator:0x7f928643e7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:18, datum_buffer:{capacity:32, datums:0x7f9287dd8390, local_datums:0x7f9287dd8390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,]}) [2024-09-26 16:00:50.681148] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.681180] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.681183] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.681191] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337650630325, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.681208] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650681078, end_time=1727337650681208, runtime=130, *this={this:0x7f9264bd6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BA-0-0, dag_ret:0, dag_status:2, start_time:1727337650680363, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.681216] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=853, dag_cnt=23, dag_cnts_[dag.get_type()]=23, &dag=0x7f9264c42710, dag={ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BA-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650680363, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.681224] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BA-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650680357, task_id:YB42C0A8002D-0006230056D8C6BA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.681243] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337650681241, task_id:YB42C0A8002D-0006230056D8C6BC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.681246] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=3] schedule one task(task=0x7f9264c43530, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.681278] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.681282] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.681284] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:50.681289] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.681292] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.681301] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.681322] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337650630342, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.681336] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9272839330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.681352] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=12] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.681355] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.681361] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9272839330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101065}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.681383] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101065}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9272839330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101065}, table_count:1, [{i:0, table_key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650680947, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=412us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.681409] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BB-0-0, dag_ret:0, dag_status:2, start_time:1727337650680898, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9272839490, output_block_cnt_arr:0x7f9272839498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337650680947, estimated_finish_time:1727337667681408}) [2024-09-26 16:00:50.681416] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c43000, type:15, status:2, dag:{ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BB-0-0, dag_ret:0, dag_status:2, start_time:1727337650680898, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.681422] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=5] task finish process(ret=0, start_time=1727337650680936, end_time=1727337650681422, runtime=486, *this={this:0x7f9264c43000, type:15, status:2, dag:{ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BB-0-0, dag_ret:0, dag_status:2, start_time:1727337650680898, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.682169] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=2] schedule one task(task=0x7f926593c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.682255] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=101065, range=0x7f92ab0d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264c80060, count:6, allocator:0x7f92bc8fa7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9264c80390, local_datums:0x7f9264c80390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:50.682291] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.682330] ERROR issue_dba_error (ob_log.cpp:1875) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.682333] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.682344] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=0] partition merge iter row count(i=0, row_count=15, ghost_row_count=0, pkey={tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337650630342, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.682368] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650682193, end_time=1727337650682368, runtime=175, *this={this:0x7f926593c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BB-0-0, dag_ret:0, dag_status:2, start_time:1727337650680898, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.682380] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1482, dag_cnt=22, dag_cnts_[dag.get_type()]=22, &dag=0x7f9264c42c40, dag={ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BB-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650680898, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.682388] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BB-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650680893, task_id:YB42C0A8002D-0006230056D8C6BB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.682510] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] succeed to add sys task(task={start_time:1727337650682508, task_id:YB42C0A8002D-0006230056D8C6BD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.682516] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=5] schedule one task(task=0x7f9264c43a60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.682538] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.682542] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.682545] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:50.682549] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.682585] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2258599936, global_cache_size=16646144, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:50.682553] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077692196634026, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8653ce0, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:505, tablet_addr:[809012-1445-0-2305843645943099392-2](offset=503808,size=593,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101066}, data_tablet_id:{id:333}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077692196634026, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3243687120, row_count:2118}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8654404, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844488830410752-0](offset=483328,size=1378,type=2,seq=0), type:"MAJOR", tablet_id:{id:101066}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843931558405474-0](offset=484706,size=859,type=2,seq=0), type:"MINI", tablet_id:{id:101066}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337650630376, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdd4f0, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221023266}, this:0x7f92a3bdd4f0, timestamp:1727334221023266, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:84, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265274283037001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696471040-2](offset=491520,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843189602812236-0](offset=491852,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879838196-0](offset=492020,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5020, 0x7f92a3bdd4f0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.682600] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=46] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.682606] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.682613] INFO [COMMON] wash (ob_kvcache_store.cpp:335) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash memory, (tenant_id=1, cache_size=16646144, lower_mem_limit=2147483648, upper_mem_limit=2147483648, min_wash_size=0, max_wash_size=0, mem_usage=1309880320, reserve_mem=137180449, wash_size=2258599936) [2024-09-26 16:00:50.682645] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337650630359, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.682654] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.682676] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.682680] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.682694] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash time detail, (compute_wash_size_time=55, refresh_score_time=14, wash_time=10) [2024-09-26 16:00:50.682688] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:333}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.682722] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=29] succeed to build merge ctx(tablet_id={id:333}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:333}, table_count:1, [{i:0, table_key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8651da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650681272, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.41ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.682756] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BC-0-0, dag_ret:0, dag_status:2, start_time:1727337650681246, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3490, output_block_cnt_arr:0x7f92728f3498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337650681272, estimated_finish_time:1727337667682755}) [2024-09-26 16:00:50.682763] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c43530, type:15, status:2, dag:{ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BC-0-0, dag_ret:0, dag_status:2, start_time:1727337650681246, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.682768] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=4] task finish process(ret=0, start_time=1727337650681256, end_time=1727337650682767, runtime=1511, *this={this:0x7f9264c43530, type:15, status:2, dag:{ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BC-0-0, dag_ret:0, dag_status:2, start_time:1727337650681246, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.682836] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=4] schedule one task(task=0x7f9264d763c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.682877] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=333, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:7, allocator:0x7f92636967d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f92664f03b0, local_datums:0x7f92664f03b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:50.683029] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.683067] ERROR issue_dba_error (ob_log.cpp:1875) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.683070] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.683082] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=0] partition merge iter row count(i=0, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337650630359, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.683101] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650682847, end_time=1727337650683101, runtime=254, *this={this:0x7f9264d763c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BC-0-0, dag_ret:0, dag_status:2, start_time:1727337650681246, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.683111] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1864, dag_cnt=21, dag_cnts_[dag.get_type()]=21, &dag=0x7f9264c43170, dag={ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BC-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650681246, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.683118] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BC-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650681241, task_id:YB42C0A8002D-0006230056D8C6BC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.683137] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337650683135, task_id:YB42C0A8002D-0006230056D8C6BE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.683140] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=3] schedule one task(task=0x7f9264c43f90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.683155] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337650630376, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.683166] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.683178] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.683177] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.683180] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.683181] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.683184] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:50.683188] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.683191] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.683185] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101066}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.683197] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.683203] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101066}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101066}, table_count:1, [{i:0, table_key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8653ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650682532, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=651us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.683245] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=30] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BD-0-0, dag_ret:0, dag_status:2, start_time:1727337650682516, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db7490, output_block_cnt_arr:0x7f9287db7498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337650682532, estimated_finish_time:1727337667683244}) [2024-09-26 16:00:50.683251] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c43a60, type:15, status:2, dag:{ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BD-0-0, dag_ret:0, dag_status:2, start_time:1727337650682516, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.683256] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=5] task finish process(ret=0, start_time=1727337650682531, end_time=1727337650683256, runtime=725, *this={this:0x7f9264c43a60, type:15, status:2, dag:{ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BD-0-0, dag_ret:0, dag_status:2, start_time:1727337650682516, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.683275] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=2] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1132452, clean_num=31457) [2024-09-26 16:00:50.683291] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=4] schedule one task(task=0x7f9264be6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.683329] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=101066, range=0x7f92a9a56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:7}, cols_desc:{ObIArray:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:7, allocator:0x7f92636867d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f92664f03b0, local_datums:0x7f92664f03b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:50.683492] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.683527] ERROR issue_dba_error (ob_log.cpp:1875) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.683530] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.683539] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=1] partition merge iter row count(i=0, row_count=182, ghost_row_count=0, pkey={tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337650630376, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.683558] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650683302, end_time=1727337650683558, runtime=256, *this={this:0x7f9264be6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BD-0-0, dag_ret:0, dag_status:2, start_time:1727337650682516, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.683567] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1051, dag_cnt=20, dag_cnts_[dag.get_type()]=20, &dag=0x7f9264c436a0, dag={ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BD-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650682516, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.683575] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BD-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650682508, task_id:YB42C0A8002D-0006230056D8C6BD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.683598] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337650683597, task_id:YB42C0A8002D-0006230056D8C6BF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.683604] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=6] schedule one task(task=0x7f9264c444c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.683601] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337650630393, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.683632] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=30] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9265b01330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=19, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.683672] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.683676] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.683679] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101009, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dd4060, count:4, allocator:0x7f928643e7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287dd4350, local_datums:0x7f9287dd4350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:50.683681] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9265b01330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:334}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:21, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.683714] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.683701] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:334}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9265b01330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:334}, table_count:1, [{i:0, table_key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8654c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650683163, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=515us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.683744] ERROR issue_dba_error (ob_log.cpp:1875) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.683746] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.683755] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BE-0-0, dag_ret:0, dag_status:2, start_time:1727337650683140, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265b014a0, output_block_cnt_arr:0x7f9265b014a8, concurrent_cnt:1, estimate_row_cnt:993, estimate_occupy_size:0, latest_update_ts:1727337650683163, estimated_finish_time:1727337667683754}) [2024-09-26 16:00:50.683754] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337650630258, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.683761] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c43f90, type:15, status:2, dag:{ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BE-0-0, dag_ret:0, dag_status:2, start_time:1727337650683140, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.683766] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=5] task finish process(ret=0, start_time=1727337650683162, end_time=1727337650683766, runtime=604, *this={this:0x7f9264c43f90, type:15, status:2, dag:{ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BE-0-0, dag_ret:0, dag_status:2, start_time:1727337650683140, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.683772] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650683637, end_time=1727337650683772, runtime=135, *this={this:0x7f9287cf83c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B6-0-0, dag_ret:0, dag_status:2, start_time:1727337650677613, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.683804] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=6191, dag_cnt=19, dag_cnts_[dag.get_type()]=19, &dag=0x7f9264c41250, dag={ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B6-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650677613, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.683808] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.683811] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.683811] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B6-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650677608, task_id:YB42C0A8002D-0006230056D8C6B6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.683813] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:50.683817] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.683832] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=3] schedule one task(task=0x7f9264be6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.683837] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337650683836, task_id:YB42C0A8002D-0006230056D8C6C0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.683840] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=3] schedule one task(task=0x7f9264c449f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.683820] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077691154184001, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8650e00, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:497, tablet_addr:[809012-1445-0-2305843637353218048-2](offset=557056,size=585,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:451}, data_tablet_id:{id:451}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077691154184001, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2119849913, row_count:31}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86514b8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844404004851712-0](offset=528384,size=1299,type=2,seq=0), type:"MAJOR", tablet_id:{id:451}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563069147000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843980950574355-0](offset=529683,size=905,type=2,seq=0), type:"MINI", tablet_id:{id:451}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986382094000, upper_trans_version:1726149986382094000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337650630410, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdfc00, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334222657108}, this:0x7f92a3bdfc00, timestamp:1727334222657108, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753331112, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:89, clock:641728512}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293246986000, v:0}, rec_scn:{val:1726265293246986000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696516096-2](offset=536576,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843170275504460-0](offset=536908,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879883234-0](offset=537058,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446403268-0](offset=537284,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc4080, 0x7f92a3bdfc00, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.683871] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=50] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.683877] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.683900] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.683903] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.683904] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:50.683894] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=334, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:19, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:21}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265896060, count:8, allocator:0x7f9269ffc7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:21, datum_buffer:{capacity:32, datums:0x7f92658963e0, local_datums:0x7f92658963e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,]}) [2024-09-26 16:00:50.683907] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.683910] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864cf80, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[814042-5586-0-2305843644870074368-2](offset=1220608,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:378}, data_tablet_id:{id:378}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2166189977, row_count:350}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864d65c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844352465928192-0](offset=1212416,size=1251,type=2,seq=0), type:"MAJOR", tablet_id:{id:378}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563080818000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843914379265251-0](offset=1213667,size=843,type=2,seq=0), type:"MINI", tablet_id:{id:378}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557779816000, upper_trans_version:1726121557779816000}, {i:1, addr:[814042-5586-0-2305843914379266094-0](offset=1214510,size=843,type=2,seq=0), type:"MINI", tablet_id:{id:378}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986451466000, upper_trans_version:1726149986451466000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337650630427, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdf430, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221743022}, this:0x7f92a3bdf430, timestamp:1727334221743022, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:88, clock:641728512}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293334630000, v:0}, rec_scn:{val:1726265293334630000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513829376-2](offset=1216512,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843176718635446-0](offset=1216950,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880563282-0](offset=1217106,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447083316-0](offset=1217332,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7f2b0, 0x7f92a3bdf430, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.683942] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=32] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.683946] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.684089] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.684122] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.684125] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.684135] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=0] partition merge iter row count(i=0, row_count=153, ghost_row_count=0, pkey={tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337650630393, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.684153] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650683851, end_time=1727337650684153, runtime=302, *this={this:0x7f9264be6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BE-0-0, dag_ret:0, dag_status:2, start_time:1727337650683140, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.684162] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1022, dag_cnt=18, dag_cnts_[dag.get_type()]=18, &dag=0x7f9264c43bd0, dag={ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BE-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650683140, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.684171] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BE-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337650683135, task_id:YB42C0A8002D-0006230056D8C6BE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.684220] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337650684218, task_id:YB42C0A8002D-0006230056D8C6C1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.684223] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=3] schedule one task(task=0x7f9264c44f20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.684253] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=7][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.684256] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.684258] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:50.684261] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.684263] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.684268] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241548806527000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.685250] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337650630427, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.685261] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.685273] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.685276] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.685270] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337650630308, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.685281] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:378}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.685284] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9264c8f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=41, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.685298] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.685302] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.685298] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:378}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:378}, table_count:1, [{i:0, table_key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650683897, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.38ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.685307] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264c8f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:377}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:43, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.685319] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C0-0-0, dag_ret:0, dag_status:2, start_time:1727337650683840, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab480, output_block_cnt_arr:0x7f92684ab488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337650683897, estimated_finish_time:1727337667685319}) [2024-09-26 16:00:50.685360] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=41] succeed to init merge ctx(task={this:0x7f9264c449f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C0-0-0, dag_ret:0, dag_status:2, start_time:1727337650683840, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.685326] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:377}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9264c8f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:377}, table_count:1, [{i:0, table_key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650678574, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=6.73ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.685365] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=5] task finish process(ret=0, start_time=1727337650683895, end_time=1727337650685365, runtime=1470, *this={this:0x7f9264c449f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C0-0-0, dag_ret:0, dag_status:2, start_time:1727337650683840, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.685384] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=44] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B9-0-0, dag_ret:0, dag_status:2, start_time:1727337650678012, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264c8f480, output_block_cnt_arr:0x7f9264c8f488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337650678574, estimated_finish_time:1727337667685384}) [2024-09-26 16:00:50.685392] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264c425a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B9-0-0, dag_ret:0, dag_status:2, start_time:1727337650678012, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.685391] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337650630410, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.685398] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=5] task finish process(ret=0, start_time=1727337650678573, end_time=1727337650685397, runtime=6824, *this={this:0x7f9264c425a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B9-0-0, dag_ret:0, dag_status:2, start_time:1727337650678012, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.685399] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9272839330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=14, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.685408] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.685410] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.685422] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=6] schedule one task(task=0x7f9287cf83c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.685436] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=3] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.685415] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9272839330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:451}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:16, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.685448] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=28] succeed to build merge ctx(tablet_id={id:451}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9272839330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:451}, table_count:1, [{i:0, table_key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8650e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650683803, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.61ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.685466] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BF-0-0, dag_ret:0, dag_status:2, start_time:1727337650683604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9272839470, output_block_cnt_arr:0x7f9272839478, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337650683803, estimated_finish_time:1727337667685466}) [2024-09-26 16:00:50.685471] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c444c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BF-0-0, dag_ret:0, dag_status:2, start_time:1727337650683604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.685475] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=4] task finish process(ret=0, start_time=1727337650683801, end_time=1727337650685475, runtime=1674, *this={this:0x7f9264c444c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BF-0-0, dag_ret:0, dag_status:2, start_time:1727337650683604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.685506] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752511928, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.685516] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.685518] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.685515] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=378, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dce060, count:5, allocator:0x7f92bc8f87d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9287dce360, local_datums:0x7f9287dce360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:50.685522] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:121}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752511928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.685542] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.685536] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:121}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650684250, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.27ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.685552] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C1-0-0, dag_ret:0, dag_status:2, start_time:1727337650684223, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3480, output_block_cnt_arr:0x7f92728f3488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337650684250, estimated_finish_time:1727337667685552}) [2024-09-26 16:00:50.685557] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c44f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C1-0-0, dag_ret:0, dag_status:2, start_time:1727337650684223, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:50.685561] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=4] task finish process(ret=0, start_time=1727337650684248, end_time=1727337650685561, runtime=1313, *this={this:0x7f9264c44f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C1-0-0, dag_ret:0, dag_status:2, start_time:1727337650684223, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:50.685576] ERROR issue_dba_error (ob_log.cpp:1875) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.685578] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.685588] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337650630427, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.685608] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650685447, end_time=1727337650685608, runtime=161, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C0-0-0, dag_ret:0, dag_status:2, start_time:1727337650683840, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.685617] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1777, dag_cnt=17, dag_cnts_[dag.get_type()]=17, &dag=0x7f9264c44630, dag={ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C0-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650683840, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.685624] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C0-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650683836, task_id:YB42C0A8002D-0006230056D8C6C0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.685689] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=377, range=0x7f92a9ed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:41, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:43}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dd0060, count:5, allocator:0x7f92bc8f87d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:43, datum_buffer:{capacity:43, datums:0x7f9287dd12b0, local_datums:0x7f9287dd0360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,]}) [2024-09-26 16:00:50.685729] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.685774] ERROR issue_dba_error (ob_log.cpp:1875) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.685778] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.685787] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=0] partition merge iter row count(i=0, row_count=6, ghost_row_count=0, pkey={tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337650630308, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.685806] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650685650, end_time=1727337650685806, runtime=156, *this={this:0x7f9287cf83c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B9-0-0, dag_ret:0, dag_status:2, start_time:1727337650678012, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.685816] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=7804, dag_cnt=16, dag_cnts_[dag.get_type()]=16, &dag=0x7f9264c421e0, dag={ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6B9-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650678012, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.685823] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6B9-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650678009, task_id:YB42C0A8002D-0006230056D8C6B9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.685839] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=3] schedule one task(task=0x7f92685b23c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.685844] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=4] schedule one task(task=0x7f926595a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.685851] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337650685850, task_id:YB42C0A8002D-0006230056D8C6C2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.685854] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=3] schedule one task(task=0x7f9264c45450, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.685858] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650685857, task_id:YB42C0A8002D-0006230056D8C6C3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.685860] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=2] schedule one task(task=0x7f9264c45980, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.685903] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=451, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:14, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:16}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f926592e060, count:4, allocator:0x7f92636867d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:16, datum_buffer:{capacity:32, datums:0x7f926592e350, local_datums:0x7f926592e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,]}) [2024-09-26 16:00:50.685926] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.685953] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.685956] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.685963] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337650630410, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.685978] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650685870, end_time=1727337650685978, runtime=108, *this={this:0x7f92685b23c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BF-0-0, dag_ret:0, dag_status:2, start_time:1727337650683604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.685986] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2382, dag_cnt=15, dag_cnts_[dag.get_type()]=15, &dag=0x7f9264c44100, dag={ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6BF-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650683604, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.685993] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6BF-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650683597, task_id:YB42C0A8002D-0006230056D8C6BF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.686034] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=121, range=0x7f92aad52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f926589a060, count:5, allocator:0x7f92c18207d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f926589a360, local_datums:0x7f926589a360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:50.686047] INFO [STORAGE] dump_info (ob_sstable_struct.cpp:262) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=10] dump merge info(msg="macro block builder close", output_row_per_s=0, new_macro_KB_per_s=0, *this={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:32, merge_start_time:1727337650686015, merge_finish_time:1727337650686047, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:0, max:0, sum:0, count:0}}, {type:"cost_time", info:{min:0, max:0, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:0, max:0, sum:0, count:0}}, {type:"incremental_row_count", info:{min:0, max:0, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:0, snapshot_version:0, start_scn:0, end_scn:0}, macro_id_list:"", comment:""}) [2024-09-26 16:00:50.686061] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=14] partition merge iter row count(i=0, row_count=0, ghost_row_count=0, pkey={tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, table={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337650630444, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.686071] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1544) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=9] merge macro blocks ok(idx_=0, task={this:0x7f926595a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C1-0-0, dag_ret:0, dag_status:2, start_time:1727337650684223, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:50.686080] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=6] task finish process(ret=0, start_time=1727337650686012, end_time=1727337650686080, runtime=68, *this={this:0x7f926595a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C1-0-0, dag_ret:0, dag_status:2, start_time:1727337650684223, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:50.686104] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.686108] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.686110] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:50.686114] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.686118] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.686125] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.686167] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.686170] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.686172] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:50.686175] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.686178] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.686183] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.686201] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=1] schedule one task(task=0x7f926595a250, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.686207] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337650686206, task_id:YB42C0A8002D-0006230056D8C6C4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.686210] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=3] schedule one task(task=0x7f9264c45eb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.686224] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=20][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.686228] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.686230] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:50.686232] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.686235] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.686239] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.686260] INFO [STORAGE.COMPACTION] get_merged_sstable (ob_tablet_merge_task.cpp:1285) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=7] create new merged sstable(ctx.param_.tablet_id_={id:121}, snapshot_version=1726241132764173999, ctx.param_.merge_type_=3, ctx.create_snapshot_version_=0, table_mode_flag=0) [2024-09-26 16:00:50.686269] INFO [STORAGE] close (ob_index_block_builder.cpp:815) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=4] succeed to close sstable index builder(res={root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_block_ids_.count():0, other_block_ids_.count():0, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, data_column_cnt:10, data_column_checksums:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], row_count:0, max_merged_trans_version:0, contain_uncommitted_row:false, occupy_size:0, original_size:0, data_checksum:0, use_old_macro_block_count:0, compressor_type:1, root_row_store_type:0, nested_offset:0, nested_size:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}) [2024-09-26 16:00:50.686283] INFO [STORAGE] init (ob_sstable.cpp:118) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=9] succeeded to init sstable(ret=0, this={ObITable:{this:0x7f92728f2128, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f92728f2128, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:null, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:50.686333] INFO [STORAGE.COMPACTION] create_sstable (ob_tablet_merge_ctx.cpp:367) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=49] succeed to merge sstable(param={table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_logic_seq:0, schema_version:1710828752511928, create_snapshot_version:0, progressive_merge_round:0, progressive_merge_step:0, is_ready_for_read:true, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, index_type:0, root_block_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), root_block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, root_row_store_type:0, latest_row_store_type:1, data_index_tree_height:0, data_block_macro_meta_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), data_block_macro_meta:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, use_old_macro_block_count:0, row_count:0, rowkey_column_cnt:5, column_cnt:10, column_checksums:[], data_checksum:0, occupy_size:0, original_size:0, max_merged_trans_version:1726241132764173999, ddl_scn:{val:0, v:0}, contain_uncommitted_row:false, is_meta_root:false, compressor_type:1, encrypt_id:0, master_key_id:0, recycle_version:0, nested_offset:0, nested_size:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}, table_key={tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_merge_info={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:1797, merge_start_time:1727337650684250, merge_finish_time:1727337650686047, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"cost_time", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"incremental_row_count", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:1, snapshot_version:0, start_scn:1726211867305630000, end_scn:1726241548806527000}, macro_id_list:"", comment:""}) [2024-09-26 16:00:50.686402] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337650630461, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.686413] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752725472, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752725472, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.686436] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=19] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.686473] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=36] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.686481] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:215}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752725472, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.686495] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:215}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752725472, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:215}, table_count:1, [{i:0, table_key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650686099, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=380us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.686513] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C2-0-0, dag_ret:0, dag_status:2, start_time:1727337650685854, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505460, output_block_cnt_arr:0x7f9266505468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337650686099, estimated_finish_time:1727337667686512}) [2024-09-26 16:00:50.686534] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=20] succeed to init merge ctx(task={this:0x7f9264c45450, type:15, status:2, dag:{ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C2-0-0, dag_ret:0, dag_status:2, start_time:1727337650685854, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.686539] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=5] task finish process(ret=0, start_time=1727337650686097, end_time=1727337650686539, runtime=442, *this={this:0x7f9264c45450, type:15, status:2, dag:{ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C2-0-0, dag_ret:0, dag_status:2, start_time:1727337650685854, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.686568] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337650630496, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.686577] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.686587] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.686589] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.686594] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:154}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.686609] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:154}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:154}, table_count:1, [{i:0, table_key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8623ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650686220, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=372us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.686627] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C4-0-0, dag_ret:0, dag_status:2, start_time:1727337650686210, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab480, output_block_cnt_arr:0x7f92684ab488, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337650686220, estimated_finish_time:1727337667686627}) [2024-09-26 16:00:50.686632] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c45eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C4-0-0, dag_ret:0, dag_status:2, start_time:1727337650686210, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.686637] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=5] task finish process(ret=0, start_time=1727337650686219, end_time=1727337650686637, runtime=418, *this={this:0x7f9264c45eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C4-0-0, dag_ret:0, dag_status:2, start_time:1727337650686210, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.686656] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337650630478, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.686664] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753355744, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753355744, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.686673] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.686675] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.686679] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:456}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753355744, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.686710] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=27] succeed to build merge ctx(tablet_id={id:456}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753355744, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:456}, table_count:1, [{i:0, table_key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8634c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650686163, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=515us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.686730] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C3-0-0, dag_ret:0, dag_status:2, start_time:1727337650685860, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515490, output_block_cnt_arr:0x7f9268515498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337650686163, estimated_finish_time:1727337667686729}) [2024-09-26 16:00:50.686735] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c45980, type:15, status:2, dag:{ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C3-0-0, dag_ret:0, dag_status:2, start_time:1727337650685860, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.686740] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=4] task finish process(ret=0, start_time=1727337650686162, end_time=1727337650686739, runtime=577, *this={this:0x7f9264c45980, type:15, status:2, dag:{ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C3-0-0, dag_ret:0, dag_status:2, start_time:1727337650685860, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.686808] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=2] schedule one task(task=0x7f9264d763c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.686817] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=7] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.686822] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=3] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.686929] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=456, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265894060, count:7, allocator:0x7f92bc8f87d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f92658943b0, local_datums:0x7f92658943b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:50.687033] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.687036] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.687045] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=0] partition merge iter row count(i=0, row_count=45, ghost_row_count=0, pkey={tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337650630478, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.687066] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650686894, end_time=1727337650687066, runtime=172, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C3-0-0, dag_ret:0, dag_status:2, start_time:1727337650685860, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.687076] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1215, dag_cnt=14, dag_cnts_[dag.get_type()]=14, &dag=0x7f9264c455c0, dag={ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C3-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650685860, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.687084] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C3-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650685857, task_id:YB42C0A8002D-0006230056D8C6C3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.687135] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=215, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:6}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fe060, count:3, allocator:0x7f92864407d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92665fe320, local_datums:0x7f92665fe320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:50.687158] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.687187] ERROR issue_dba_error (ob_log.cpp:1875) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.687190] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.687198] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=0] partition merge iter row count(i=0, row_count=9, ghost_row_count=0, pkey={tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337650630461, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.687214] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650687109, end_time=1727337650687213, runtime=104, *this={this:0x7f9264d763c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C2-0-0, dag_ret:0, dag_status:2, start_time:1727337650685854, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.687221] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1367, dag_cnt=13, dag_cnts_[dag.get_type()]=13, &dag=0x7f9264c45090, dag={ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C2-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650685854, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.687228] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C2-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650685850, task_id:YB42C0A8002D-0006230056D8C6C2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.687269] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=154, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:5, allocator:0x7f92865fa7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665f8360, local_datums:0x7f92665f8360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:50.687383] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.687388] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.687396] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337650630496, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.687414] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650687247, end_time=1727337650687414, runtime=167, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C4-0-0, dag_ret:0, dag_status:2, start_time:1727337650686210, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.687432] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1222, dag_cnt=12, dag_cnts_[dag.get_type()]=12, &dag=0x7f9264c45af0, dag={ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C4-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650686210, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.687439] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C4-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650686206, task_id:YB42C0A8002D-0006230056D8C6C4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.687476] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337650687474, task_id:YB42C0A8002D-0006230056D8C6C5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.687480] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=3] schedule one task(task=0x7f9264c463e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.687486] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337650687484, task_id:YB42C0A8002D-0006230056D8C6C6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.687489] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=3] schedule one task(task=0x7f9264c46910, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.687496] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337650687495, task_id:YB42C0A8002D-0006230056D8C6C7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.687498] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=2] schedule one task(task=0x7f9264c46e40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.687531] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.687535] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.687537] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:50.687541] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.687540] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.687545] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.687546] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.687549] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:50.687552] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.687554] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.687558] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.687565] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.687583] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.687586] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.687587] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:50.687590] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.687594] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b863eec0, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:499, tablet_addr:[816017-1849-0-2305843639502065664-2](offset=1921024,size=587,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101016}, data_tablet_id:{id:154}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726074677203981000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:398898191, row_count:578}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844419038609408-0](offset=1900544,size=1313,type=2,seq=0), type:"MAJOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726074015354550000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844024975361313-0](offset=1901857,size=946,type=2,seq=0), type:"MINOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:1726074677203981000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726205802467242000, upper_trans_version:1726205802467242000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337650630530, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd5fc0, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209508808}, this:0x7f92a3bd5fc0, timestamp:1727334209508808, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:69, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277803107894000, v:0}, rec_scn:{val:1726241804194470000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771625984-2](offset=1904640,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161774413-0](offset=1904973,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881251311-0](offset=1905135,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc7f00, 0x7f92a3bd5fc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.687636] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=42] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.687640] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.687716] INFO [STORAGE] build_new_table_store (ob_tablet_table_store.cpp:1206) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=22] succeed to build new table store(major_tables_={cnt:1, is_inited:true}, minor_tables_={cnt:3, is_inited:true}, memtables_={count:1}, PRINT_TS(*this)={ObTabletTableStore_Pretty:this:0x7f92ab1cfca8, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, memtables:{count:1}, is_ready_for_read:true, table_array:{ [2851] [ YB42C0A8002D-0006230056D8C6C1-0-0 ] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [2851] [ YB42C0A8002D-0006230056D8C6C1-0-0 ] MAJOR MAJOR 1725120003420849000 1725120003420849000 {val:0, v:0} {val:1726077600415496000, v:0} 0 false [2851] [ YB42C0A8002D-0006230056D8C6C1-0-0 ] MINOR MINOR 1726178585269925002 1726178585269925002 {val:1726077259159067000, v:0} {val:1726178585269925002, v:0} 0 false [2851] [ YB42C0A8002D-0006230056D8C6C1-0-0 ] MINI 1726211867106157000 1726211867106157000 {val:1726178585269925002, v:0} {val:1726211867305630000, v:0} 0 false [2851] [ YB42C0A8002D-0006230056D8C6C1-0-0 ] MINI 1726241132764173999 1726241132764173999 {val:1726211867305630000, v:0} {val:1726241548806527000, v:0} 0 false [2851] [ YB42C0A8002D-0006230056D8C6C1-0-0 ] MEM ACTIVE 9223372036854775807 18446744073709547609 {val:1726241548806527000, v:0} {val:4611686018427387903, v:0} 3 unused }}) [2024-09-26 16:00:50.689264] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337650630513, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.689278] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.689293] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.689296] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.689302] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101015}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.689321] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101015}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101015}, table_count:1, [{i:0, table_key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650687516, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.78ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.689349] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C5-0-0, dag_ret:0, dag_status:2, start_time:1727337650687479, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515490, output_block_cnt_arr:0x7f9268515498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337650687516, estimated_finish_time:1727337667689348}) [2024-09-26 16:00:50.689355] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c463e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C5-0-0, dag_ret:0, dag_status:2, start_time:1727337650687479, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.689349] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337650630547, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.689361] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=5] task finish process(ret=0, start_time=1727337650687514, end_time=1727337650689361, runtime=1847, *this={this:0x7f9264c463e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C5-0-0, dag_ret:0, dag_status:2, start_time:1727337650687479, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.689365] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.689382] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=12] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.689386] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.689382] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337650630530, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.689390] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9265a13330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.689399] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.689401] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.689392] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101063}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.689405] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9265a13330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101016}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.689412] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:101063}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101063}, table_count:1, [{i:0, table_key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650687533, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.86ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.689419] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:101016}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9265a13330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101016}, table_count:1, [{i:0, table_key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650687572, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.83ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.689439] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C6-0-0, dag_ret:0, dag_status:2, start_time:1727337650687489, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265a13490, output_block_cnt_arr:0x7f9265a13498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337650687572, estimated_finish_time:1727337667689439}) [2024-09-26 16:00:50.689444] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c46910, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C6-0-0, dag_ret:0, dag_status:2, start_time:1727337650687489, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.689449] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=5] task finish process(ret=0, start_time=1727337650687571, end_time=1727337650689449, runtime=1878, *this={this:0x7f9264c46910, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C6-0-0, dag_ret:0, dag_status:2, start_time:1727337650687489, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.689469] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C7-0-0, dag_ret:0, dag_status:2, start_time:1727337650687498, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665054a0, output_block_cnt_arr:0x7f92665054a8, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337650687533, estimated_finish_time:1727337667689469}) [2024-09-26 16:00:50.689489] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=19] succeed to init merge ctx(task={this:0x7f9264c46e40, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C7-0-0, dag_ret:0, dag_status:2, start_time:1727337650687498, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.689493] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=2] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.689495] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=5] task finish process(ret=0, start_time=1727337650687530, end_time=1727337650689494, runtime=1964, *this={this:0x7f9264c46e40, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C7-0-0, dag_ret:0, dag_status:2, start_time:1727337650687498, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.689499] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=4] schedule one task(task=0x7f9264c49260, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.689573] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=101016, range=0x7f92aa6d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c6e060, count:6, allocator:0x7f92bc8f87d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9262c6e390, local_datums:0x7f9262c6e390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:50.689623] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.689661] ERROR issue_dba_error (ob_log.cpp:1875) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.689664] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.689673] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337650630530, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.689694] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650689541, end_time=1727337650689694, runtime=153, *this={this:0x7f9264c49260, type:1, status:2, dag:{ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C6-0-0, dag_ret:0, dag_status:2, start_time:1727337650687489, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.689703] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2214, dag_cnt=11, dag_cnts_[dag.get_type()]=11, &dag=0x7f9264c46550, dag={ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C6-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650687489, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.689711] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C6-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650687484, task_id:YB42C0A8002D-0006230056D8C6C6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.689759] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101015, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265898060, count:6, allocator:0x7f92bc8f87d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9265898390, local_datums:0x7f9265898390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:50.689820] ERROR issue_dba_error (ob_log.cpp:1875) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.689825] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.689833] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337650630513, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.689848] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650689734, end_time=1727337650689848, runtime=114, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C5-0-0, dag_ret:0, dag_status:2, start_time:1727337650687479, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.689855] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2376, dag_cnt=10, dag_cnts_[dag.get_type()]=10, &dag=0x7f9264c46020, dag={ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C5-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650687479, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.689861] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C5-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650687474, task_id:YB42C0A8002D-0006230056D8C6C5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.689888] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=1] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.689895] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337650689894, task_id:YB42C0A8002D-0006230056D8C6C8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.689899] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=3] schedule one task(task=0x7f9264c47370, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.689902] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337650689902, task_id:YB42C0A8002D-0006230056D8C6C9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.689904] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=2] schedule one task(task=0x7f9264c478a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.689955] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101063, range=0x7f92ab152200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b08060, count:8, allocator:0x7f9269ffc7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b083e0, local_datums:0x7f9265b083e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:50.689974] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.690004] ERROR issue_dba_error (ob_log.cpp:1875) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.690004] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.690006] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=1][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.690010] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.690013] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:50.690018] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.690013] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=0] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337650630547, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.690032] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650689916, end_time=1727337650690032, runtime=116, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C7-0-0, dag_ret:0, dag_status:2, start_time:1727337650687498, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.690039] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2541, dag_cnt=9, dag_cnts_[dag.get_type()]=9, &dag=0x7f9264c46a80, dag={ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C7-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650687498, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.690045] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C7-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650687495, task_id:YB42C0A8002D-0006230056D8C6C7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.690023] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=4] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706139416032, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f926e072bc0, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[812168-2208-0-2305843644869693440-2](offset=839680,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:328}, data_tablet_id:{id:328}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706139416032, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:867269498, row_count:786}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f926e07329c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[812168-2208-0-2305844589762490368-0](offset=831488,size=1472,type=2,seq=0), type:"MAJOR", tablet_id:{id:328}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562828712000}}, minor_tables:{{i:0, addr:[812168-2208-0-2305843935853721024-0](offset=832960,size=863,type=2,seq=0), type:"MINI", tablet_id:{id:328}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986144852000, upper_trans_version:1726149986144852000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337650630580, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bda610, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220621826}, this:0x7f92a3bda610, timestamp:1727334220621826, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:78, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265257450254001, v:0}, rec_scn:{val:1726264907253071001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[812168-2208-0-2305843366770556928-2](offset=835584,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[812168-2208-0-2305843176718254413-0](offset=835917,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843251880182249-0](offset=836073,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843216446702283-0](offset=836299,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc8ea0, 0x7f92a3bda610, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.690069] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=46] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.690072] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.690076] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.690076] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.690078] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:50.690083] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.690102] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.690110] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.690155] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650690153, task_id:YB42C0A8002D-0006230056D8C6CA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.690159] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=4] schedule one task(task=0x7f9264c47dd0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.690176] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.690179] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.690181] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:50.690185] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.690187] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.690192] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.691015] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337650630563, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.691030] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=14] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9272839330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=15, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.691043] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.691046] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.691051] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9272839330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:452}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:17, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.691070] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:452}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9272839330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:452}, table_count:1, [{i:0, table_key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8652d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650690067, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=981us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.691109] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=33] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C8-0-0, dag_ret:0, dag_status:2, start_time:1727337650689898, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9272839480, output_block_cnt_arr:0x7f9272839488, concurrent_cnt:1, estimate_row_cnt:10, estimate_occupy_size:0, latest_update_ts:1727337650690067, estimated_finish_time:1727337667691109}) [2024-09-26 16:00:50.691116] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c47370, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C8-0-0, dag_ret:0, dag_status:2, start_time:1727337650689898, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.691110] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337650630580, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.691121] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=5] task finish process(ret=0, start_time=1727337650690066, end_time=1727337650691121, runtime=1055, *this={this:0x7f9264c47370, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C8-0-0, dag_ret:0, dag_status:2, start_time:1727337650689898, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.691127] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=16] update column cnt(column_cnt_=26, store_column_cnt_=26, column_cnt_=26, column_array_.count()=0) [2024-09-26 16:00:50.691131] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=4] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92664f1480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=26, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.691140] INFO [MDS] init_for_evict_medium_info (ob_tablet_mds_data.cpp:414) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=13] succeeded to init mds data(ret=0, finish_medium_scn=1726077600415496000, src_medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, src_extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}) [2024-09-26 16:00:50.691149] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.691153] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.691158] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92664f1480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:328}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:28, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.691178] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:328}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92664f1480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:328}, table_count:1, [{i:0, table_key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e072bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650689998, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.16ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.691204] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C9-0-0, dag_ret:0, dag_status:2, start_time:1727337650689904, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92664f15d0, output_block_cnt_arr:0x7f92664f15d8, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337650689998, estimated_finish_time:1727337667691204}) [2024-09-26 16:00:50.691229] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=24] succeed to init merge ctx(task={this:0x7f9264c478a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C9-0-0, dag_ret:0, dag_status:2, start_time:1727337650689904, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.691235] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=6] task finish process(ret=0, start_time=1727337650689995, end_time=1727337650691235, runtime=1240, *this={this:0x7f9264c478a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C9-0-0, dag_ret:0, dag_status:2, start_time:1727337650689904, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.691158] INFO [STORAGE] init_for_merge (ob_tablet.cpp:363) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=7] succeeded to init tablet for mini/minor/major merge(ret=0, param={sstable:0x7f92728f2128, snapshot_version:1726241132764173999, clog_checkpoint_scn:{val:1726241548806527000, v:0}, multi_version_start:1726178585269925002, need_report:false, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, rebuild_seq:0, update_with_major_flag:false, need_check_sstable:false, ddl_info:{keep_old_ddl_sstable:true, ddl_start_scn:{val:0, v:0}, ddl_commit_scn:{val:0, v:0}, ddl_checkpoint_scn:{val:0, v:0}, ddl_snapshot_version:0, ddl_execution_id:-1, data_format_version:0}, allow_duplicate_sstable:false, merge_type:"MINI_MERGE", need_check_transfer_seq:true, transfer_seq:0}, old_tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:3, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337650630444, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}, this={this:0x7f92684b0060, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x7f92ab1d0450, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726241548806527000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726241132764173999, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92684b0670, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}, {i:2, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726241132764173999, upper_trans_version:1726241132764173999}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x7f92684b0520, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f92684b1880, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f92684b1950, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{redefined:false, snapshot_version:0, schema_version:0, data_tablet_id:{id:0}, hidden_tablet_id:{id:0}, lob_meta_tablet_id:{id:50121}, lob_piece_tablet_id:{id:60121}, is_old_mds:true}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:50.691264] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=91] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9268514060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}}, ref_cnt:0, upper_trans_version:1725120003420849000, timestamp:0}, this:0x7f9268514060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1725120003420849000, max_merged_trans_version:1725120003420849000, data_macro_block_count:1, nested_size:4096, nested_offset:589824, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:84, row_count_:14, occupy_size_:1604, original_size_:1914, data_checksum_:3587602083, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:1, data_micro_block_count_:1, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:2, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1725120003420849000, max_merged_trans_version_:1725120003420849000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9268514310, column_checksum_count:10, data_root_info:{addr:[0-0-0-3458764933653594112-0](offset=0,size=391,type=3,seq=0), block_data:{buf:0x7f9268514360, size:391, extra_buf:0x7f92685144e7, extra_size:424, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-3458765009889263616-0](offset=0,size=462,type=3,seq=0), block_data:{buf:0x7f926851468f, size:462, extra_buf:null, extra_size:0, type:1}}, data_block_count:1, other_block_count:0, linked_block_count:0, data_block_ids:0x7f926851485d, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:true, nested_offset:589824, nested_size:4096}}, valid_for_reading:true}) [2024-09-26 16:00:50.691285] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=19] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f92684b4060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}}, ref_cnt:0, upper_trans_version:1726178585269925002, timestamp:0}, this:0x7f92684b4060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726178585269925002, max_merged_trans_version:1726178585269925002, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726178585269925002, v:0}, meta:{basic_meta:{version:1, length:94, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726178585269925002, max_merged_trans_version_:1726178585269925002, recycle_version:1726077600415496000, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726178585269925002, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:1, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f92684b4310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:50.691296] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=10] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9268514060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}}, ref_cnt:0, upper_trans_version:1726211867106157000, timestamp:0}, this:0x7f9268514060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726211867106157000, max_merged_trans_version:1726211867106157000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726211867305630000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726211867106157000, max_merged_trans_version_:1726211867106157000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726211867305630000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9268514310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:50.691306] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=9] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f92684b4060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f92684b4060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f92684b4310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:50.691317] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=9][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.691352] ERROR issue_dba_error (ob_log.cpp:1875) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.691355] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf9de5b1 0xf9dcfd6 0xe51c6fb 0xe519c03 0xe518f89 0xe516f8c 0xf01158d 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.691374] INFO [STORAGE] reset (ob_tablet.cpp:159) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=1] reset tablet(this=0x7f92684b0060, ls_id={id:1}, tablet_id={id:121}, lbt()="0x121f1f9c 0xe52cc09 0xe52c6c5 0xe52bd13 0x4fd348f 0xf01110c 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:50.691382] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1250) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=0][errcode=-4184] sstable merge finish(ret=-4184, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C1-0-0, dag_ret:0, dag_status:2, start_time:1727337650684223, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3480, output_block_cnt_arr:0x7f92728f3488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337650684250, estimated_finish_time:1727337667685552}, compaction_filter:NULL, time_guard:total=2.10ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, task={this:0x7f926595a250, type:16, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C1-0-0, dag_ret:0, dag_status:2, start_time:1727337650684223, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:50.691407] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650686259, end_time=1727337650691407, runtime=5148, *this={this:0x7f926595a250, type:16, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C1-0-0, dag_ret:0, dag_status:2, start_time:1727337650684223, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:50.691468] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=7245, dag_cnt=8, dag_cnts_[dag.get_type()]=8, &dag=0x7f9264c44b60, dag={ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C1-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650684223, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}) [2024-09-26 16:00:50.691476] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C1-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650684218, task_id:YB42C0A8002D-0006230056D8C6C1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.691502] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337650630597, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.691511] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=9] update column cnt(column_cnt_=12, store_column_cnt_=12, column_cnt_=12, column_array_.count()=0) [2024-09-26 16:00:50.691514] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9265a13490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.691533] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.691536] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.691541] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265a13490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:330}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.691560] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:330}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9265a13490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:330}, table_count:1, [{i:0, table_key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e107fe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650690172, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.37ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.691590] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CA-0-0, dag_ret:0, dag_status:2, start_time:1727337650690159, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265a135f0, output_block_cnt_arr:0x7f9265a135f8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337650690172, estimated_finish_time:1727337667691590}) [2024-09-26 16:00:50.691596] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c47dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CA-0-0, dag_ret:0, dag_status:2, start_time:1727337650690159, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.691602] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=6] task finish process(ret=0, start_time=1727337650690170, end_time=1727337650691602, runtime=1432, *this={this:0x7f9264c47dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CA-0-0, dag_ret:0, dag_status:2, start_time:1727337650690159, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.691629] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=3] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.691636] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=6] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.691640] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=3] schedule one task(task=0x7f9264d763c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.691645] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650691643, task_id:YB42C0A8002D-0006230056D8C6CB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.691648] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=3] schedule one task(task=0x7f9264c48300, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.691659] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:50.691664] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.691726] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=452, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:15, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:17}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dd2060, count:5, allocator:0x7f926368e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:17, datum_buffer:{capacity:32, datums:0x7f9287dd2360, local_datums:0x7f9287dd2360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,]}) [2024-09-26 16:00:50.691813] ERROR issue_dba_error (ob_log.cpp:1875) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.691817] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.691827] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=0] partition merge iter row count(i=0, row_count=10, ghost_row_count=0, pkey={tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337650630563, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.691849] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650691683, end_time=1727337650691849, runtime=166, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C8-0-0, dag_ret:0, dag_status:2, start_time:1727337650689898, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.691858] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1960, dag_cnt=7, dag_cnts_[dag.get_type()]=7, &dag=0x7f9264c46fb0, dag={ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C8-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650689898, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.691865] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C8-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650689894, task_id:YB42C0A8002D-0006230056D8C6C8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.691912] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=328, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:26, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:28}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:5, allocator:0x7f926368e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:28, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,]}) [2024-09-26 16:00:50.691942] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.691971] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.691973] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.691981] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337650630580, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.691996] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650691887, end_time=1727337650691996, runtime=109, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C9-0-0, dag_ret:0, dag_status:2, start_time:1727337650689904, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.692003] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2099, dag_cnt=6, dag_cnts_[dag.get_type()]=6, &dag=0x7f9264c474e0, dag={ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6C9-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650689904, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.692009] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6C9-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650689902, task_id:YB42C0A8002D-0006230056D8C6C9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.692057] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=330, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265894060, count:7, allocator:0x7f926368e7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f92658943b0, local_datums:0x7f92658943b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:50.692221] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.692255] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.692258] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.692267] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=0] partition merge iter row count(i=0, row_count=141, ghost_row_count=0, pkey={tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337650630597, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.692285] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650692029, end_time=1727337650692285, runtime=256, *this={this:0x7f9264d763c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CA-0-0, dag_ret:0, dag_status:2, start_time:1727337650690159, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.692294] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2135, dag_cnt=5, dag_cnts_[dag.get_type()]=5, &dag=0x7f9264c47a10, dag={ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CA-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650690159, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.692301] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CA-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650690153, task_id:YB42C0A8002D-0006230056D8C6CA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.692329] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.692332] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.692335] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:50.692340] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.692343] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.692351] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.692421] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337650692419, task_id:YB42C0A8002D-0006230056D8C6CC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.692425] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=4] schedule one task(task=0x7f9264c48830, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.692431] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337650692430, task_id:YB42C0A8002D-0006230056D8C6CD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.692433] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=2] schedule one task(task=0x7f9264c48d60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.692449] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.692452] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.692453] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:50.692457] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.692460] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8648b00, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[816017-1849-0-2305843650239631360-2](offset=2068480,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:332}, data_tablet_id:{id:332}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:235179760, row_count:171}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8649200, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844652040736768-0](offset=2052096,size=1530,type=2,seq=0), type:"MAJOR", tablet_id:{id:332}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843956256036346-0](offset=2053626,size=882,type=2,seq=0), type:"MINI", tablet_id:{id:332}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}, {i:1, addr:[816017-1849-0-2305843965919713644-0](offset=2054508,size=891,type=2,seq=0), type:"MINI", tablet_id:{id:332}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726208018709503000, upper_trans_version:1726208018709503000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337650630631, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdbd80, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220854400}, this:0x7f92a3bdbd80, timestamp:1727334220854400, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:81, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265274283037001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479514673152-2](offset=2060288,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161930166-0](offset=2060726,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881407064-0](offset=2060888,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447927098-0](offset=2061114,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bcade0, 0x7f92a3bdbd80, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.692543] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=83] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.692549] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.692615] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.692618] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:50.692620] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:50.692623] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.692626] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b86a2d40, is_inited:true, wash_score:1727337650630105000, hold_ref_cnt:true, ref_cnt:2, version:2, length:511, tablet_addr:[816017-1849-0-2305843652387110912-2](offset=2064384,size=599,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101067}, data_tablet_id:{id:334}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1854554341, row_count:13383}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86a3488, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844595132416000-0](offset=2048000,size=1477,type=2,seq=0), type:"MAJOR", tablet_id:{id:101067}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843951961064901-0](offset=2049477,size=878,type=2,seq=0), type:"MINI", tablet_id:{id:101067}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}, {i:1, addr:[816017-1849-0-2305843958403516723-0](offset=2050355,size=884,type=2,seq=0), type:"MINI", tablet_id:{id:101067}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726208018709503000, upper_trans_version:1726208018709503000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337650630648, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bde490, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221276956}, this:0x7f92a3bde490, timestamp:1727334221276956, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:86, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265292880560000, v:0}, rec_scn:{val:1726264918599518000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843482735894528-2](offset=2056192,size=441,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046827961-0](offset=2056633,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881402983-0](offset=2056807,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bcbd80, 0x7f92a3bde490, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:50.692683] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.692687] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.692718] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337650630614, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.692726] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=8] update column cnt(column_cnt_=29, store_column_cnt_=29, column_cnt_=29, column_array_.count()=0) [2024-09-26 16:00:50.692729] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9265a13490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.692744] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.692747] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.692752] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9265a13490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:329}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.692785] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=27] succeed to build merge ctx(tablet_id={id:329}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9265a13490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:329}, table_count:1, [{i:0, table_key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e089f20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650692323, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=427us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.692811] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CB-0-0, dag_ret:0, dag_status:2, start_time:1727337650691648, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9265a135f0, output_block_cnt_arr:0x7f9265a135f8, concurrent_cnt:1, estimate_row_cnt:62, estimate_occupy_size:0, latest_update_ts:1727337650692323, estimated_finish_time:1727337667692811}) [2024-09-26 16:00:50.692818] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c48300, type:15, status:2, dag:{ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CB-0-0, dag_ret:0, dag_status:2, start_time:1727337650691648, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.692824] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=5] task finish process(ret=0, start_time=1727337650692321, end_time=1727337650692824, runtime=503, *this={this:0x7f9264c48300, type:15, status:2, dag:{ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CB-0-0, dag_ret:0, dag_status:2, start_time:1727337650691648, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.692842] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=2] schedule one task(task=0x7f9287cf83c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.692918] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=329, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dd4060, count:6, allocator:0x7f92636867d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9287dd4390, local_datums:0x7f9287dd4390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:50.693047] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.693082] ERROR issue_dba_error (ob_log.cpp:1875) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.693085] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.693095] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=1] partition merge iter row count(i=0, row_count=47, ghost_row_count=0, pkey={tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337650630614, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.693115] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650692854, end_time=1727337650693114, runtime=260, *this={this:0x7f9287cf83c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CB-0-0, dag_ret:0, dag_status:2, start_time:1727337650691648, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.693124] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1475, dag_cnt=4, dag_cnts_[dag.get_type()]=4, &dag=0x7f9264c47f40, dag={ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CB-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650691648, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.693131] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CB-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650691643, task_id:YB42C0A8002D-0006230056D8C6CB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.693198] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337650630631, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.693207] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=25, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.693219] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.693223] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.693228] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:332}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:27, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.693247] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:332}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:332}, table_count:1, [{i:0, table_key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8648b00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650692445, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=781us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.693268] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CC-0-0, dag_ret:0, dag_status:2, start_time:1727337650692425, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515490, output_block_cnt_arr:0x7f9268515498, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337650692445, estimated_finish_time:1727337667693268}) [2024-09-26 16:00:50.693274] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c48830, type:15, status:2, dag:{ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CC-0-0, dag_ret:0, dag_status:2, start_time:1727337650692425, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.693280] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=5] task finish process(ret=0, start_time=1727337650692444, end_time=1727337650693279, runtime=835, *this={this:0x7f9264c48830, type:15, status:2, dag:{ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CC-0-0, dag_ret:0, dag_status:2, start_time:1727337650692425, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.693293] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=4] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.693345] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=332, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:25, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:27}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dce060, count:6, allocator:0x7f92864427d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:27, datum_buffer:{capacity:32, datums:0x7f9287dce390, local_datums:0x7f9287dce390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,]}) [2024-09-26 16:00:50.693405] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.693408] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.693416] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337650630631, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.693432] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337650693304, end_time=1727337650693432, runtime=128, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CC-0-0, dag_ret:0, dag_status:2, start_time:1727337650692425, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.693439] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1014, dag_cnt=3, dag_cnts_[dag.get_type()]=3, &dag=0x7f9264c48470, dag={ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CC-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650692425, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.693446] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CC-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337650692419, task_id:YB42C0A8002D-0006230056D8C6CC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.693468] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337650630648, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:50.693476] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:50.693517] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=30] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:50.693520] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:50.693526] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101067}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:50.693544] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101067}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101067}, table_count:1, [{i:0, table_key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b86a2d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337650692610, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=914us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:50.693571] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CD-0-0, dag_ret:0, dag_status:2, start_time:1727337650692433, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db34a0, output_block_cnt_arr:0x7f9287db34a8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337650692610, estimated_finish_time:1727337667693571}) [2024-09-26 16:00:50.693577] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c48d60, type:15, status:2, dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CD-0-0, dag_ret:0, dag_status:2, start_time:1727337650692433, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.693582] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=5] task finish process(ret=0, start_time=1727337650692608, end_time=1727337650693582, runtime=974, *this={this:0x7f9264c48d60, type:15, status:2, dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CD-0-0, dag_ret:0, dag_status:2, start_time:1727337650692433, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.693595] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=3] schedule one task(task=0x7f9264bd6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:50.693664] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=40] multi version scan iterator init succ(param.table_id_=101067, range=0x7f92aae56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265896060, count:8, allocator:0x7f92636867d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92658963e0, local_datums:0x7f92658963e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:50.693835] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=19][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.693875] ERROR issue_dba_error (ob_log.cpp:1875) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.693878] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.693888] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=0] partition merge iter row count(i=0, row_count=227, ghost_row_count=0, pkey={tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337650630648, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.693909] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650693630, end_time=1727337650693909, runtime=279, *this={this:0x7f9264bd6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CD-0-0, dag_ret:0, dag_status:2, start_time:1727337650692433, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.693921] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1487, dag_cnt=2, dag_cnts_[dag.get_type()]=2, &dag=0x7f9264c489a0, dag={ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CD-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650692433, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.693929] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CD-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337650692430, task_id:YB42C0A8002D-0006230056D8C6CD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.718489] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:50.721746] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:50.721756] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.723500] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [2925][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=8] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f92b77d02b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-09-26 16:00:50.730718] INFO [STORAGE] write_meta_block (ob_index_block_builder.cpp:1808) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=16] succeed to write macro meta in macro block(ret=0, macro_meta_={val:{version:1, length:156, data_checksum:2073194008, rowkey_count:4, rowkey_count:4, column_count:4, micro_block_count:144, occupy_size:2055851, data_size:7595160, data_zsize:2046419, original_size:7595160, progressive_merge_round:6, block_offset:2055851, block_size:7155, row_count:124307, row_count_delta:-47188, max_merged_trans_version:1726190662817616000, is_encrypted:false, is_deleted:false, contain_uncommitted_row:false, compressor_type:2, master_key_id:0, encrypt_id:0, encrypt_key:"", row_store_type:0, schema_version:1711680543204904, snapshot_version:1726241132764173999, is_last_row_last_flag:false, logic_id:{data_seq:0, logic_version:1726241290462324999, tablet_id:1152921504606847223}, macro_id:[1091](ver=0,mode=0,seq=824628), column_checksums:[0, 0, 0, 0], has_string_out_row:false, all_lob_in_row:false}, end_key:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9265b0a140, hex: 3230323430393133303232303238, cstr: 20240913022028},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9265b0a14e, hex: 343132323136313031, cstr: 412216101},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0a0e0, hex: 686266CBA26D0BE8, int: -1726165486201839000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0a118, hex: 0000000000000000, int: 0},]store_rowkey:}}) [2024-09-26 16:00:50.731022] INFO [STORAGE.BLKMGR] async_write (ob_macro_block_handle.cpp:178) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=33] Async write macro block(macro_id_=[1091](ver=0,mode=0,seq=824628)) [2024-09-26 16:00:50.731031] INFO [STORAGE] flush (ob_macro_block.cpp:864) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=8] macro block writer succeed to flush macro block.(block_id=[1091](ver=0,mode=0,seq=824628), common_header_={header_size:24, version:1, magic:1001, attr:1, payload_size:2063243, payload_checksum:-359285756}, macro_header_={fixed_header:{header_size:192, version:2, magic:1007, tablet_id:1152921504606847223, logical_version:1726241290462324999, data_seq:0, column_count:4, rowkey_column_count:4, row_store_type:0, row_count:124307, occupy_size:2055851, micro_block_count:144, micro_block_data_offset:216, micro_block_data_size:2055635, idx_block_offset:2055851, idx_block_size:7155, meta_block_offset:2063006, meta_block_size:261, data_checksum:2073194008, compressor_type:2, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", col_type_array_cnt:4}, column_types:0x7f92b72040e8, column_orders:0x7f92b72040f8, column_checksum:0x7f92b7204108, column_checksum:[0, 0, 0, 0]}, contain_uncommitted_row=false, max_merged_trans_version=1726190662817616000, ¯o_handle=0x7f9264b21e28) [2024-09-26 16:00:50.731050] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.731129] ERROR issue_dba_error (ob_log.cpp:1875) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.731140] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=11][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1bc0 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.731153] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=0] partition merge iter row count(i=0, row_count=4508, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.731180] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=21] partition merge iter row count(i=1, row_count=119763, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337650629912, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.731213] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650643554, end_time=1727337650731212, runtime=87658, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A4-0-0, dag_ret:0, dag_status:2, start_time:1727337650631010, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.731232] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=100222, dag_cnt=1, dag_cnts_[dag.get_type()]=1, &dag=0x7f9264c3b4f0, dag={ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A4-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650631010, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.731244] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A4-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337650631007, task_id:YB42C0A8002D-0006230056D8C6A4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.736046] INFO [STORAGE] write_meta_block (ob_index_block_builder.cpp:1808) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=4] succeed to write macro meta in macro block(ret=0, macro_meta_={val:{version:1, length:156, data_checksum:1746792960, rowkey_count:4, rowkey_count:4, column_count:4, micro_block_count:144, occupy_size:2053874, data_size:8207786, data_zsize:2044442, original_size:8207786, progressive_merge_round:6, block_offset:2053874, block_size:7310, row_count:134087, row_count_delta:-47820, max_merged_trans_version:1726190662817616000, is_encrypted:false, is_deleted:false, contain_uncommitted_row:false, compressor_type:2, master_key_id:0, encrypt_id:0, encrypt_key:"", row_store_type:0, schema_version:1711680543204904, snapshot_version:1726241132764173999, is_last_row_last_flag:false, logic_id:{data_seq:0, logic_version:1726241290462324999, tablet_id:1152921504606847222}, macro_id:[1812](ver=0,mode=0,seq=824629), column_checksums:[0, 0, 0, 0], has_string_out_row:false, all_lob_in_row:false}, end_key:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f926589e140, hex: 3230323430393133303232323034, cstr: 20240913022204},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f926589e14e, hex: 343132333337363234, cstr: 412337624},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f926589e0e0, hex: 50DCFA50B36D0BE8, int: -1726165415241262000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f926589e118, hex: 0000000000000000, int: 0},]store_rowkey:}}) [2024-09-26 16:00:50.736354] INFO [STORAGE.BLKMGR] async_write (ob_macro_block_handle.cpp:178) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=27] Async write macro block(macro_id_=[1812](ver=0,mode=0,seq=824629)) [2024-09-26 16:00:50.736375] INFO [STORAGE] flush (ob_macro_block.cpp:864) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=20] macro block writer succeed to flush macro block.(block_id=[1812](ver=0,mode=0,seq=824629), common_header_={header_size:24, version:1, magic:1001, attr:1, payload_size:2061421, payload_checksum:45797750}, macro_header_={fixed_header:{header_size:192, version:2, magic:1007, tablet_id:1152921504606847222, logical_version:1726241290462324999, data_seq:0, column_count:4, rowkey_column_count:4, row_store_type:0, row_count:134087, occupy_size:2053874, micro_block_count:144, micro_block_data_offset:216, micro_block_data_size:2053658, idx_block_offset:2053874, idx_block_size:7310, meta_block_offset:2061184, meta_block_size:261, data_checksum:1746792960, compressor_type:2, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", col_type_array_cnt:4}, column_types:0x7f926fa040e8, column_orders:0x7f926fa040f8, column_checksum:0x7f926fa04108, column_checksum:[0, 0, 0, 0]}, contain_uncommitted_row=false, max_merged_trans_version=1726190662817616000, ¯o_handle=0x7f9263a37e28) [2024-09-26 16:00:50.736392] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:50.736452] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:50.736459] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=7][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1bc0 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:50.736471] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=0] partition merge iter row count(i=0, row_count=4249, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.736498] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=20] partition merge iter row count(i=1, row_count=130147, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337650629895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:50.736529] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337650646255, end_time=1727337650736529, runtime=90274, *this={this:0x7f9264d76030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A3-0-0, dag_ret:0, dag_status:2, start_time:1727337650631001, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:50.736545] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=105544, dag_cnt=0, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264c3afc0, dag={ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6A3-0-0, dag_ret:-4184, dag_status:5, start_time:1727337650631001, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:50.736558] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6A3-0-0] [lt=9] succeed to del sys task(removed_task={start_time:1727337650630998, task_id:YB42C0A8002D-0006230056D8C6A3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:50.743054] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:50.743088] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337650743051) [2024-09-26 16:00:50.743094] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337650642956, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:50.743125] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] get wrs ts(ls_id={id:1}, delta=223379979543, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.743135] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:50.743162] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:50.743167] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:50.743169] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:50.743176] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.743178] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.743184] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337650743101) [2024-09-26 16:00:50.754771] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:50.754797] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.768482] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768493] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=8] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60476}, multi_version_start=1726077614845725000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077614845725000, old_tablet_snaphsot_version=1726077689155434000, tablet={this:0x7f92b8655c20, is_inited:true, wash_score:-7496034386089229807, hold_ref_cnt:true, ref_cnt:2, version:2, length:476, tablet_addr:[674007-5161-0-2305843614805131264-0](offset=1048576,size=564,type=2,seq=0), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:60476}, data_tablet_id:{id:476}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726077689155434000, multi_version_start:1726077614845725000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:0, row_count:0}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86562b4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:0}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[674007-5161-0-2305843315231137792-0](offset=1024000,size=285,type=2,seq=0), type:"MAJOR", tablet_id:{id:60476}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077600415496000}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{null}}}, addr:[674007-5161-0-2305843251880390656-0](offset=1044480,size=226,type=2,seq=0)}, storage_schema_addr:{ptr:null, addr:[674007-5161-0-2305843163833561314-0](offset=1044706,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[674007-5161-0-2305843251880391026-0](offset=1044850,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0}) [2024-09-26 16:00:50.768530] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=36] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60476}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768535] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768538] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200426}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768541] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768543] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60379}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768546] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768550] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50380}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768554] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768555] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200048}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768559] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768560] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847407}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768564] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200450}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768569] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768570] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200197}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768574] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768575] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:233}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768580] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768582] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200127}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768585] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768587] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50114}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768590] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768592] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50192}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768595] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768597] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200251}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768600] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768602] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60350}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768605] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768607] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60429}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768610] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768613] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60259}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768616] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768618] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847145}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768621] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768623] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:188}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768626] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768628] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847104}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768631] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768632] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:224}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768636] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768637] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:308}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768640] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768644] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:369}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768647] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768649] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200325}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768652] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768654] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:493}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768657] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768659] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101011}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768662] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768664] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847061}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768666] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768668] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846994}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768671] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768674] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50124}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768678] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768680] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200124}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768683] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768685] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:284}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768688] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768690] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50149}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768693] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768695] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:307}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768698] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768700] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50240}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768703] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768706] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50355}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768709] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768711] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60457}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768714] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768716] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200312}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768719] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768721] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:109}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768724] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768726] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:182}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768730] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768732] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101098}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768734] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768737] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60387}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768741] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768743] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847037}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768746] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768748] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847198}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768751] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768752] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60297}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768755] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768757] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60305}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768761] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768763] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60342}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768766] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768770] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768773] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768775] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847084}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768778] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768780] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847417}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768784] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768786] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101036}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768789] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768790] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200107}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768794] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768795] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847359}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768800] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768802] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60005}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768806] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768807] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60130}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768810] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768812] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847232}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768815] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768817] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200301}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768821] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768823] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101064}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768826] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768827] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60129}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768831] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768834] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:404}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768837] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768839] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200130}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768842] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768844] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50393}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768847] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768848] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101080}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768852] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768854] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200052}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768857] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768858] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200063}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768862] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768865] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50424}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768868] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768870] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200031}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768873] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768875] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200314}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768878] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768880] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:197}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768883] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768885] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50381}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768888] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768890] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200492}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768893] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768896] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:249}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768899] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768901] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200332}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768904] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768906] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:103}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768909] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768911] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50373}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768914] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768916] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200304}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768919] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768920] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200300}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768923] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768926] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846997}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768930] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768932] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847013}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768935] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768937] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50437}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768940] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768942] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200206}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768945] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768947] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101083}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768950] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768952] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847294}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768955] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768958] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200433}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768961] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768963] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101062}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768966] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768968] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60263}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768971] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768973] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200480}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768976] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768977] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:152}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768981] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768982] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847199}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768985] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768987] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:129}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768991] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768993] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50364}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.768996] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.768998] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200364}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769001] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769003] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50279}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769006] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769008] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101018}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769011] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769013] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60167}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769016] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769017] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60424}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769022] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769024] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60438}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769027] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769029] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60440}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769032] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769034] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847055}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769036] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769038] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101034}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769041] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769043] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200262}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769046] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769048] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847006}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769052] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769054] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:255}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769057] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769059] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101084}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769062] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769064] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50219}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769067] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769069] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60233}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769072] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769074] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200086}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769077] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769079] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50142}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769082] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769085] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200163}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769088] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769090] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50120}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769093] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769095] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50406}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769098] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769100] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50315}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769103] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769104] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200183}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769107] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769109] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847002}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769112] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769115] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101068}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769118] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769120] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60140}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769123] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769125] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847081}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769128] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769130] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200136}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769133] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769135] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:439}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769138] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769139] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50484}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769142] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769144] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200109}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769149] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769151] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:165}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769154] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769156] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:157}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769159] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769160] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60145}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769163] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769165] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847155}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769168] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769170] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:363}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769173] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769175] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847345}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769179] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769181] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200310}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769184] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769186] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60444}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769189] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769191] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50329}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769194] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769196] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60226}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769199] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769201] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:225}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769204] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769205] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200330}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769210] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769212] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60441}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769215] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769217] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847357}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769220] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769222] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847046}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769225] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769227] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200448}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769229] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769231] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847099}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769234] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769236] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847363}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769241] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769243] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60253}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769246] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769248] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101073}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769251] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769253] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200178}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769256] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769257] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847349}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769260] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769262] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60383}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769265] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769267] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:128}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769270] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769273] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200034}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769276] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769278] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:427}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769281] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769283] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:294}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769286] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769288] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769291] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769293] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847242}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769296] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769297] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200122}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769300] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769302] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:392}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769307] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769309] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50379}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769312] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769313] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50450}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769316] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769318] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200493}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769321] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769323] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:196}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769326] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769327] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60243}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769330] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769332] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200184}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769337] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769340] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847362}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769355] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=12] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769357] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:349}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769361] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769363] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60166}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769378] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769381] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847228}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769384] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769386] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847289}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769389] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769391] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847394}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769394] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769396] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60484}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769400] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769402] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200168}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769405] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769458] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=10] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847378}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769471] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=7] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769476] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50334}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769479] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769481] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101031}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769485] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769487] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60459}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769490] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769492] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846983}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769495] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769497] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60425}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769500] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769502] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50408}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769505] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769507] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60349}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769510] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769512] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847418}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769515] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769517] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846990}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769520] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769522] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847049}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769524] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769526] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60113}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769529] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769531] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847416}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769534] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769536] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60352}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769539] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769541] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847057}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769544] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769546] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847200}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769548] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769550] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60180}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769553] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769555] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:473}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769559] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769560] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50325}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769563] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769565] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847105}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769568] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769570] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200021}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769573] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769575] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847187}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769577] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769579] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60431}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769582] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769584] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50292}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769587] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769588] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60410}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769591] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769593] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:372}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769596] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769598] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50140}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769601] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769603] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60491}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769606] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769607] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769611] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769614] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:429}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769620] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769623] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60165}, multi_version_start=1726077614845725000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769626] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769628] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200171}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769631] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769633] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101005}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769636] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769638] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:362}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769644] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769646] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200025}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769649] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769651] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:405}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769657] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769659] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60309}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769662] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769664] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101027}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769667] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769669] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200204}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769673] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769675] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846986}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769679] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769681] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:210}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769687] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769689] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847402}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769693] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769694] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60103}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769709] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769714] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:00:50.769718] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769719] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50246}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769723] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769725] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50251}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769729] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769731] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846977}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769735] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769737] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847101}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769742] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769744] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847036}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769747] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769750] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200038}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769753] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769756] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847344}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769761] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769763] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60262}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769767] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769769] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847282}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769773] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769775] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50003}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769778] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769780] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200491}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769784] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769786] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60279}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769790] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769792] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60355}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769795] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769798] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60354}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769802] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769804] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60488}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769809] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769812] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:381}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769815] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769817] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60329}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769820] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769822] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101082}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769825] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769828] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:344}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769831] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769833] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200464}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769837] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769839] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60430}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769844] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769846] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50263}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769850] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769851] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50115}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769855] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769857] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60104}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769863] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769865] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60245}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769868] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769870] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50239}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769875] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769877] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:432}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769880] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769882] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847110}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769885] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769887] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769891] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769893] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60478}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769896] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769899] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200346}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769902] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769905] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847157}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769908] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769912] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:327}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769916] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769918] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847281}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769932] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=12] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769935] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200215}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769939] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769941] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60248}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769947] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769949] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50109}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769952] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769954] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847364}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769958] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769959] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200419}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769963] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769964] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200009}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769967] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769970] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:410}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769973] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769976] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60147}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769979] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769981] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847076}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769985] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769987] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200337}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769990] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769992] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847292}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.769995] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.769998] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60361}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770001] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770003] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60116}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770007] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770009] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847043}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770013] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770015] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60323}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770019] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770021] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101057}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770024] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770026] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101001}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770030] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770032] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60158}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770036] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770038] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60358}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770042] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770044] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200101}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770047] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770050] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847321}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770053] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770055] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:423}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770059] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770062] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60334}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770065] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770067] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50261}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770071] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770073] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200363}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770077] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770079] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60296}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770083] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770085] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200342}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770089] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770090] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60381}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770094] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60380}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770099] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770102] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200452}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770107] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770109] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50391}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770112] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770114] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60487}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770117] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770119] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101033}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770122] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770124] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50479}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770127] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770129] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:316}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770132] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770134] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60251}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770137] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770139] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60293}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770143] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770145] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50392}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770151] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770153] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200340}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770156] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770158] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60181}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770161] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770163] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200456}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770167] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770169] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770173] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770175] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50305}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770179] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770181] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847056}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770184] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770186] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50121}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770190] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770192] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:232}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770197] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770199] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:320}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770202] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770204] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846980}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770208] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770210] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847354}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770216] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770218] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200094}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770221] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770223] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50401}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770226] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770228] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:499}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770232] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770234] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60119}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770238] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770240] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50004}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770245] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770248] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50475}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770251] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770252] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:166}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770256] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770258] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101030}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770261] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770263] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847111}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770267] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770269] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200247}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770273] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770275] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101094}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770279] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770281] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101020}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770284] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770287] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50306}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770290] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770292] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50117}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770297] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770300] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200451}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770303] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770305] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101044}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770309] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770311] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847406}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770314] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770317] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847272}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770320] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770322] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847346}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770335] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=11] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770338] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847217}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770342] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770344] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:100002}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770347] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770350] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60452}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770356] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770358] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200200}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770361] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770363] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200460}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770367] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770369] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60386}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770372] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770375] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60133}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770378] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770380] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50167}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770383] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770386] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50440}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770389] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770391] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:292}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770395] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770397] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:293}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770400] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770402] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200040}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770406] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770408] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50395}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770411] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770414] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847396}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770417] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770420] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:290}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770423] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770425] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:118}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770429] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770431] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50146}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770434] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770436] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50207}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770440] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770442] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200228}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770447] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770450] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:371}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770453] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770456] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847191}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770459] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770461] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200454}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770465] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770469] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847086}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770472] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770474] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847039}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770477] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770480] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847381}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770484] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770486] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847139}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770489] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770491] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:388}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770497] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770499] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847365}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770502] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770503] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60450}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770506] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770509] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200020}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770512] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770514] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:445}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770518] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770520] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200141}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770524] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770526] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101017}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770530] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770532] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847278}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770535] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770537] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50455}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770540] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770543] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847370}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770548] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770550] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847375}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770554] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770555] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200030}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770559] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770561] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101013}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770564] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:192}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770570] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770572] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770575] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770578] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60294}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770581] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770583] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200045}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770587] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770589] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60214}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770592] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770596] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847077}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770600] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770602] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60398}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770606] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770608] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:401}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770611] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770614] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200485}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770617] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770620] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847033}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770623] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770626] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60256}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770630] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770633] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770636] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770638] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101072}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770642] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770644] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50458}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770647] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770650] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60432}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770653] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770656] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50431}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770661] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770663] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101088}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770667] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770669] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50150}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770672] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770674] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101069}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770678] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770680] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847286}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770683] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770686] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847087}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770690] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770693] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847015}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770696] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770699] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847420}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770702] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770704] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:244}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770707] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770709] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847270}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770715] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770717] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847361}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770720] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770734] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=13] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200296}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770738] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770740] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50357}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770743] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770744] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50245}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770748] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770749] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60493}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770752] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770754] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50410}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770757] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770759] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60351}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770762] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770764] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847154}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770770] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770772] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60106}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770775] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770777] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200066}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770780] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770783] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60402}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770786] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770788] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60108}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770792] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770794] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:191}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770798] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770800] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200209}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770803] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770805] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847118}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770809] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770811] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847014}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770814] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770816] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50249}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770820] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770822] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200350}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770825] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770827] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50128}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770831] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770833] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101060}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770837] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770839] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200069}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770842] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770846] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200210}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770850] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770852] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50256}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770855] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770858] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:252}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770862] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770865] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200229}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770870] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770872] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:279}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770876] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770877] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200037}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770893] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=14] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770895] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:296}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770899] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770901] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200422}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770907] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770909] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60336}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770912] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770914] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:357}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770918] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770921] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50442}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770927] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770929] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101022}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770932] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770934] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:151}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770938] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770941] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50362}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770944] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770947] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50110}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770950] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770952] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200462}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770956] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770958] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200302}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770964] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770966] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50252}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770969] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770971] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50471}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770974] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770977] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:325}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770982] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770984] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50226}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770987] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770989] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200423}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770993] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.770995] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50488}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.770998] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771000] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50214}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771004] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771006] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60412}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771009] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771011] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771015] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771017] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847414}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771021] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771023] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:305}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771026] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771029] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847250}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771034] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771036] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847029}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771039] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771053] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=13] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50428}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771058] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771060] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50113}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771063] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771066] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200437}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771069] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771071] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60451}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771075] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771077] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200177}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771080] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771083] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60404}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771086] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771089] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847372}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771092] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771094] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200486}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771098] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771100] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50441}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771103] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771106] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:389}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771111] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771113] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:5}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771117] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771118] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101035}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771121] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771124] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60327}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771127] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771129] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847151}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771133] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771135] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50145}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771139] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771141] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60426}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771145] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771147] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101058}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771152] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771155] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:407}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771158] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771159] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847141}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771163] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771165] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101081}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771171] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771173] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:245}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771176] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771178] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101086}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771181] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771183] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200111}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771187] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771189] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101052}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771193] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771195] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50133}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771198] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771201] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101019}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771204] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771206] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60157}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771210] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771212] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847291}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771216] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771218] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:403}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771221] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771223] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200064}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771229] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771231] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101079}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771234] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771236] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50307}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771241] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771244] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50272}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771247] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771248] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:383}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771251] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771254] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847403}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771259] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771261] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:238}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771270] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771273] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200430}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771276] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771278] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60306}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771281] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771283] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50333}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771287] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771289] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847188}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771292] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771295] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:150}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771304] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771308] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847285}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771312] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771314] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200220}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771317] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771320] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50238}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771323] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771326] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:406}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771329] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847011}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771335] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771337] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771341] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771343] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847156}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771346] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771348] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60330}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771352] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771354] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50319}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771358] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771360] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50377}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771363] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771365] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847273}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771370] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771927] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:50.771938] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=6] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771941] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:396}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771944] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771946] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200205}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771949] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771953] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60320}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771958] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771960] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200196}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771963] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771965] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200494}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771971] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771973] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60407}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771976] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771978] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200487}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771982] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771984] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847293}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771987] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771990] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50430}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771993] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.771995] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200010}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.771999] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772001] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50104}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772005] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772007] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50425}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772011] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772013] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50480}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772016] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772018] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200154}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772024] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772026] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60390}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772029] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772031] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50436}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772034] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772036] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50160}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772042] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772044] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200120}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772047] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772049] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:474}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772053] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772055] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60393}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772059] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772061] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847051}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772064] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772067] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50451}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772070] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772072] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50366}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772076] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772078] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847279}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772082] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772084] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60148}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772090] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772092] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200137}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772095] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772097] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200006}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772100] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772102] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60472}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772106] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772108] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60004}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772112] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772114] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:391}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772117] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772120] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:426}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772123] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772125] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:348}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772128] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772133] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50385}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772136] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772138] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847180}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772141] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772144] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50438}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772147] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772150] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200311}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772153] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772155] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200159}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772161] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772163] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847007}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772166] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772168] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200051}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772170] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772173] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:235}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772176] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772178] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200316}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772184] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772186] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50336}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772189] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772191] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847153}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772194] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772196] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200453}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772199] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772202] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60401}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772205] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772207] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:397}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772211] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772213] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:110}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772217] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772219] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200211}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772222] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772225] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200216}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772230] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772232] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60254}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772236] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772237] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:202}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772240] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772243] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200092}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772246] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772248] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60146}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772252] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772254] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:319}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772257] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772260] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50423}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772263] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772265] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:385}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772268] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772271] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60111}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772281] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=8] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772283] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50208}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772287] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772292] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200068}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772295] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772297] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200074}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772300] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772304] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847373}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772308] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772310] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50342}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772315] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772318] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50223}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772321] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772323] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60244}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772329] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60439}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772334] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772336] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200157}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772340] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772341] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50452}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772347] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772349] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:155}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772352] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772354] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50309}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772357] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772360] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200112}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772363] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772365] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60003}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772368] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772371] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200438}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772374] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772376] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50349}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772381] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772383] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847422}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772387] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772389] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50446}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772392] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772395] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:253}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772398] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772400] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50125}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772404] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772406] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:382}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772409] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772413] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846979}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772416] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772418] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847024}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772422] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772424] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847129}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772427] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772430] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200036}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772433] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772435] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60344}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772439] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772442] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50157}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772446] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772449] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:350}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772454] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772456] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200110}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772459] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772461] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847367}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772477] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=14] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772479] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847283}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772482] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772487] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60149}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772491] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772492] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772495] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772497] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50189}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772501] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772503] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:291}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772506] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772508] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200165}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772514] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772516] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50492}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772520] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772521] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60471}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772525] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772526] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200175}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772530] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772532] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772536] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772538] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60332}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772542] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772544] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60405}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772547] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772550] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60150}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772555] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772557] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200128}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772560] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772562] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60213}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772565] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772567] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50188}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772572] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772575] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60375}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772578] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772579] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200018}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772583] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772585] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847121}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772589] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772591] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50403}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772594] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772596] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101043}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772599] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772602] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772605] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772607] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:315}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772611] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772613] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50378}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772616] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772618] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847380}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772622] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772626] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50225}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772629] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772631] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50409}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772634] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772637] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200125}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772640] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772643] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200338}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772646] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772648] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200429}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772651] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772654] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:167}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772657] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772659] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60121}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772662] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772664] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:149}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772668] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772670] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200365}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772674] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772676] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200115}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772679] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772681] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200303}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772685] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772687] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200129}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772690] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772692] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847275}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772695] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772698] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200263}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772703] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772705] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60151}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772708] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772710] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50232}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772714] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772716] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200182}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772719] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772721] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200444}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772725] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772727] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847280}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772730] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772733] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200024}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772736] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772738] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50236}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772742] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772744] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:143}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772749] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772751] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:189}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772754] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772756] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846988}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772759] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772761] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:446}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772764] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772767] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60246}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772770] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772772] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101032}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772776] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772778] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60345}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772782] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772784] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:208}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772787] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772789] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200008}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772792] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772794] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847350}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772797] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772800] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50356}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772803] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772805] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:124}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772808] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772811] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:326}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772814] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772816] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60455}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772819] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772821] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50331}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772824] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772827] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200428}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772830] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772832] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200005}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772836] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772838] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200208}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772842] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772844] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50457}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772847] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772849] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200308}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772852] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772854] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50332}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772858] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772860] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60128}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772864] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772866] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50472}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772869] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772872] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50474}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772875] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772877] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200449}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772881] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772883] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50248}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772887] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772889] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:107}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772892] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772895] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200173}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772898] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772900] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50324}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772904] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772906] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50432}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772910] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772912] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60168}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772916] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772917] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:102}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772921] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772923] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60392}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772927] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772929] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50296}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772933] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772935] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60236}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772939] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772941] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:478}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772944] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772946] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50294}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772949] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772952] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50358}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772955] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772958] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847097}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772961] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772963] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50360}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772969] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772971] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772974] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772975] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200001}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772978] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772981] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200118}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772984] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772986] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101028}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772990] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772993] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:376}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.772997] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.772999] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:239}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773003] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773004] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847054}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773008] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773010] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200139}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773013] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773015] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50253}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773019] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773021] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773026] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773028] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50132}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773032] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773034] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60378}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773037] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773039] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200138}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773042] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773045] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:411}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773048] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773050] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50155}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773055] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773057] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101021}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773061] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773063] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60324}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773067] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773069] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50343}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773072] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773074] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50250}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773078] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773080] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847271}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773083] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773086] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847295}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773090] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773093] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200484}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773096] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773098] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200160}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773104] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773106] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50487}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773109] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773111] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60292}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773114] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773116] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60196}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773120] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773123] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50350}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773126] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773127] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60239}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773130] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773133] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200071}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773139] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773141] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50498}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773144] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773146] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:111}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773149] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773151] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847063}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773154] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773156] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847053}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773160] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773162] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60416}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773166] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773168] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50111}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773171] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773174] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:477}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773178] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773181] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:209}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773196] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773199] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60366}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773203] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773205] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60369}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773209] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773211] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200468}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773214] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773216] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847424}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773220] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773222] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60110}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773225] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773228] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200258}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773230] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773233] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50116}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773236] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773238] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:443}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773241] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773244] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50375}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773247] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773249] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200082}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773261] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=9] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773264] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847140}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773266] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773268] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101037}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773272] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773274] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847117}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773277] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773280] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:390}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773283] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773285] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:263}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773289] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773291] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:135}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773294] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773296] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:335}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773300] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773302] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101054}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773305] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773307] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60363}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773310] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773313] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200465}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773316] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773318] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50444}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773322] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773324] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:153}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773327] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773330] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847080}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773332] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773335] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200472}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773338] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773340] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50165}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773344] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773346] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847098}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773349] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773352] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847415}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773355] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773357] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60395}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773360] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773362] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60364}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773365] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773367] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50119}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773371] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773373] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847160}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773376] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773379] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60433}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773382] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773384] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:374}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773388] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773390] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50005}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773394] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773396] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847412}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773399] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773401] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50123}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773405] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773407] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101041}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773411] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773413] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200436}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773416] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773418] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200246}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773422] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773424] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50369}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773427] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773429] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200445}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773432] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773435] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60370}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773438] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773440] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200133}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773443] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773445] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847341}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773450] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773453] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847423}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773456] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773459] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200050}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773461] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773464] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50399}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773467] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773469] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200313}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773473] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773475] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200424}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773479] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773482] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200070}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773485] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773488] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60210}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773491] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773493] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200269}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773496] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773499] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101024}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773503] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773505] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50147}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773512] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773514] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847004}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773529] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773532] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:236}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773536] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773538] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50316}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773542] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773543] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:237}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773547] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773549] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60225}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773553] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773555] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60131}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773558] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773560] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200123}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773564] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847027}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773569] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773572] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50190}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773577] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773579] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847060}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773582] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773584] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60423}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773587] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773589] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:483}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773595] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773597] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50439}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773600] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773602] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60391}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773605] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773608] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101055}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773610] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773613] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50400}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773616] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773619] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60357}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773622] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773624] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:113}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773628] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773630] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200416}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773633] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773636] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60458}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773639] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773641] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:123}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773646] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773648] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847028}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773651] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773653] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50365}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773656] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773658] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50290}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773662] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773664] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847128}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773667] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773669] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200151}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773673] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773674] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60437}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773678] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773680] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847351}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773683] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773685] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50433}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773691] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773693] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50135}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773696] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773698] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200425}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773700] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773702] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200155}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773705] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773707] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847181}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773710] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773711] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:471}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773714] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773716] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:450}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773719] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773722] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773725] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773727] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50376}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773731] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773733] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200169}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773736] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773738] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60397}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773742] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773744] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:375}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773747] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773749] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847152}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773753] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773755] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:3}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773758] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773760] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60360}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773764] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773768] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60112}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773771] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773773] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:219}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773776] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773781] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60333}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773784] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773786] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:168}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773789] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773791] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200076}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773794] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773796] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846985}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773801] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773803] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:141}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773807] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773809] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847253}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773811] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773814] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200061}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773817] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773819] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:475}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773823] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773825] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60399}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773829] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773831] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101040}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773834] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773836] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:259}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773841] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773843] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847083}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773846] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773848] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846978}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773852] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773854] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50491}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773857] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773859] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773863] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773865] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:248}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773868] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773870] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847038}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773874] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773877] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200176}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773881] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773883] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847089}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773886] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773888] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200043}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773892] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773894] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200078}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773897] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773901] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847120}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773905] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773907] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60153}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773923] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=14] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773926] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50386}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773929] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773932] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101047}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773935] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773938] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847127}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773941] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773943] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200150}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773946] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773948] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847016}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773952] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773954] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200343}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773957] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773960] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50416}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773963] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773966] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50265}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773969] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773967] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:50.773972] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847356}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773975] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773976] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200318}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773980] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773982] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200333}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773985] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773987] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:106}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773991] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773993] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60307}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.773997] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.773999] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50411}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774002] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774004] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200238}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774008] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774009] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200482}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774013] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774015] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200257}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774019] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774021] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606846993}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774024] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774026] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50459}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774030] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774032] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50389}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774038] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774040] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200442}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774043] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774045] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101059}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774047] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774050] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200463}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774053] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774055] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50131}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774058] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774061] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200093}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774064] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774066] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50262}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774069] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774072] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200455}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774074] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774077] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:324}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774082] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774084] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847419}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774087] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774089] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847358}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774091] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774094] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:441}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774097] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774099] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60219}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774104] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774106] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:130}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774109] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774111] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200481}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774114] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774116] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:262}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774121] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774123] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847413}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774126] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774128] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50143}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774131] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774134] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60155}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774139] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774141] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200248}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774144] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774146] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200320}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774149] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774151] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:114}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774154] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774157] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50374}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774160] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774162] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60240}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774165] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774168] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:206}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774170] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774173] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:207}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774176] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774178] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:498}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774183] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774185] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:380}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774188] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774190] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200149}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774193] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774195] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774199] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774201] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60413}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774204] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774206] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847018}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774210] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774212] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50345}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774215] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774218] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200072}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774220] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774223] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200181}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774226] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774246] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200341}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774253] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=5] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774256] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50166}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774263] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=5] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774266] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60356}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774271] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774274] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50434}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774278] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774280] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:295}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774273] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337650328055]}) [2024-09-26 16:00:50.774284] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774287] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847290}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774291] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774293] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60371}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774297] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774299] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50180}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774303] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774306] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60343}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774309] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774312] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200339}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774315] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774317] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200164}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774321] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774323] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60143}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774327] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774331] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60406}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774334] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774336] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847233}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774339] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774341] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847257}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774344] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774346] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:370}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774350] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774352] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60255}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774355] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774358] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847158}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774363] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774365] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50242}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774368] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774370] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:436}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774373] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774375] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847042}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774379] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774381] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101046}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774386] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774388] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60427}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774391] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774394] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60265}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774397] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774398] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:430}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774402] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774404] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847048}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774407] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774410] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60105}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774415] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774417] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60252}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774421] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774423] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:272}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774428] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774431] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847062}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774434] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774445] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:117}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774452] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=5] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774454] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200089}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774469] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=13] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774472] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50295}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774476] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774478] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50382}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774481] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774484] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50394}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774487] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774489] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200174}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774492] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774494] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50152}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774498] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774500] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847274}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774503] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774505] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200434}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774509] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774511] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60475}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774514] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774517] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200254}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774520] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774524] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200443}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774527] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774529] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:251}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774532] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774534] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50490}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774537] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774540] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847047}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774543] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774546] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847360}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774549] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774551] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847252}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774554] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774556] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60191}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774560] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774562] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:223}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774565] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774568] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200490}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774571] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774573] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50291}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774577] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774579] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101045}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774582] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774584] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50371}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774588] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774590] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847382}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774593] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774595] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:488}, multi_version_start=1726210103162544000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774599] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774601] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847277}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774605] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774607] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200019}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774610] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774612] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60331}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774616] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774620] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847003}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774623] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774625] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847132}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774629] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774631] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:131}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774637] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774639] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60442}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774642] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774644] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60284}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774647] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774650] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847353}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774653] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774655] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200488}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774658] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774660] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847322}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774664] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774666] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847398}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774669] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774672] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847148}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774675] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774677] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847088}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774682] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774684] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50243}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774688] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774689] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200207}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774692] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774694] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60316}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774700] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774702] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60182}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774705] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774707] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774710] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774713] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200415}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774716] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774718] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200035}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774721] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774723] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:246}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774727] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774729] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:60499}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774732] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774735] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101078}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774738] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774740] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:148}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774743] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774746] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847143}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774751] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774753] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200033}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774756] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774758] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:408}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774761] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774763] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200114}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774767] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774769] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200326}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774772] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774774] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101049}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774780] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774782] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:247}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774785] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774789] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200172}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774792] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774795] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:365}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774797] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774799] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847045}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774802] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.774804] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200317}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.774807] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] REACH SYSLOG RATE LIMIT [frequency] [2024-09-26 16:00:50.775295] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=0] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.775299] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847079}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.775666] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=0] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.775670] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:240}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.775854] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=0] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.775857] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101025}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.776037] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=0] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.776039] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:50168}, multi_version_start=1726077624866534000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.776392] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=0] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:50.776396] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847023}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:50.776550] INFO [STORAGE] runTimerTask (ob_tenant_tablet_scheduler.cpp:174) [2946][T1_SSTableGC][T1][YB42C0A8002D-0006230058484B96-0-0] [lt=0] SSTableGCTask(cost_ts=10082) [2024-09-26 16:00:50.785038] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:50.785047] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.815348] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:50.815371] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.820252] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=21] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.820262] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.820268] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.820272] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.820277] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.820285] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.821197] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.821233] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.821249] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.823135] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.823145] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:50.828390] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:50.828400] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337650828388], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:50.828417] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337650828388], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:50.828422] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:50.828426] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:50.836791] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337650836790, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007926, add_bytes: 0 [2024-09-26 16:00:50.843713] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:50.843724] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337650843710) [2024-09-26 16:00:50.843730] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337650743101, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:50.843747] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223380080142, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.843753] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:50.843773] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:50.843778] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:50.843782] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:50.843788] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.843790] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.843792] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337650843733) [2024-09-26 16:00:50.845604] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:50.845611] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.870015] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337651798517, now=[mts=1727337649868865]) [2024-09-26 16:00:50.870026] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337651798517, now=[mts=1727337649868865], now0=[mts=1727337649868865], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:50.870042] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=15][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a311e400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500787, assoc_session_id:3221500787, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337649868623, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:50.870060] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a2319bf8, id:3221500787, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a311e400}) [2024-09-26 16:00:50.870068] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500787, *tx_desc={this:0x7f92a311e400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500787, assoc_session_id:3221500787, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337649868623, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:50.870077] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=8] start stmt(ret=-4283, auto_commit=true, session_id=3221500787, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a311e400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500787, assoc_session_id:3221500787, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337649868623, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337649868835, use_das=false, nested_level=0, session={this:0x7f92a2319bf8, id:3221500787, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a311e400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:50.870094] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:50.870099] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:50.870102] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:50.870105] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:50.870107] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:50.870111] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:50.870116] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=998401, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337651868517) [2024-09-26 16:00:50.870120] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:50.870127] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:50.870129] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:50.870131] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:50.870137] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:50.870176] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:50.870179] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:50.874917] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337650828388]}) [2024-09-26 16:00:50.875889] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:50.875903] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.883793] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2258599936, global_cache_size=16646144, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:50.883835] INFO [COMMON] wash (ob_kvcache_store.cpp:335) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash memory, (tenant_id=1, cache_size=16646144, lower_mem_limit=2147483648, upper_mem_limit=2147483648, min_wash_size=0, max_wash_size=0, mem_usage=1309880320, reserve_mem=137180449, wash_size=2258599936) [2024-09-26 16:00:50.883916] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash time detail, (compute_wash_size_time=59, refresh_score_time=16, wash_time=12) [2024-09-26 16:00:50.884540] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=16] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1163909, clean_num=31457) [2024-09-26 16:00:50.890540] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=3] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-09-26 16:00:50.890553] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=9] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=690) [2024-09-26 16:00:50.906150] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:50.906157] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.936427] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:50.936457] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=28][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.944294] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:50.944303] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337650944292) [2024-09-26 16:00:50.944307] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337650843733, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:50.944323] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223380179773, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:50.944331] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:50.944338] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:50.944343] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:50.944346] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:50.944352] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.944355] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:50.944356] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337650944310) [2024-09-26 16:00:50.966697] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:50.966717] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:50.974804] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=802128, replace_num=15728) [2024-09-26 16:00:50.974814] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:50.975532] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=38] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337650828388]}) [2024-09-26 16:00:50.983324] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:50.997036] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:50.997041] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.022529] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591E-0-0] [lt=34][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:51.022541] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591E-0-0] [lt=11][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:51.022550] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591E-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:51.022555] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591E-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:51.025536] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500792, support_ssl=false) [2024-09-26 16:00:51.025557] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221500792, sess.client_addr_="127.0.0.1:36362") [2024-09-26 16:00:51.025562] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500792, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:51.025601] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:51.025610] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2688][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:51.025657] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA2-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:51.025838] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA2-0-0] [lt=7][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:51.025845] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA2-0-0] [lt=6][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"=H3e$~sFBWqQ!MCEr18d"}, ret=-4043) [2024-09-26 16:00:51.025852] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA2-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:51.025855] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA2-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:51.025863] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA2-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:51.025886] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA2-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500792, proxy_sessid:0}) [2024-09-26 16:00:51.025890] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA2-0-0] [lt=3] mark session id unused(sessid=3221500792) [2024-09-26 16:00:51.025893] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA2-0-0] [lt=2][errcode=0] server close connection(sessid=3221500792, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:51.025896] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA2-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500792, proxy_sessid=0) [2024-09-26 16:00:51.025899] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA2-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:51.025901] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA2-0-0] [lt=1] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500792, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:51.025937] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500792, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337651025577, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:51.025944] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500792, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:51.025948] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500792, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337651025577, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:51.025952] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500792, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:51.027289] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.027313] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.044862] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:51.044870] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:51.044877] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337651044860) [2024-09-26 16:00:51.044881] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337650944310, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:51.044886] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:51.044895] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:51.044898] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:51.045195] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:51.045207] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:51.045211] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:51.045214] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:51.045218] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:51.045220] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:51.045223] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:51.045226] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:51.045227] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:51.045229] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:51.045231] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:51.045233] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:51.045236] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:51.045238] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:51.045244] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:51.045261] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=16][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:51.045263] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:51.045266] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:51.045269] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:51.045272] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:51.045275] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:51.045282] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:51.045287] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:51.045290] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:51.045291] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:51.045297] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:51.045300] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:51.045303] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885764-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:51.045321] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:51.045324] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:51.045329] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:51.045332] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337651045121, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:51.045346] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:51.045348] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:51.045374] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:51.045379] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337651045378, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=487, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:51.045386] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:51.045388] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:51.045402] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223380281769, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:51.045408] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:51.045414] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:51.045418] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:51.045422] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:51.045427] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.045429] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.045431] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337651045392) [2024-09-26 16:00:51.051960] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:51.051972] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:51.051976] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:51.051979] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:51.051982] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:51.051984] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:51.052001] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=16][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:51.052004] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:51.052005] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:51.052007] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:51.052009] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:51.052010] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:51.052012] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:51.052014] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:51.052019] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:51.052021] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:51.052023] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:51.052025] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:51.052027] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:51.052029] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:51.052031] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:51.052035] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:51.052059] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=23][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:51.052062] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:51.052064] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:51.052070] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:51.052073] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:51.052076] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:51.052079] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:51.052081] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588C-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:51.052101] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588C-0-0] [lt=19][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:51.052104] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588C-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337651051912, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:51.052108] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588C-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:51.052110] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588C-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:51.052139] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588C-0-0] [lt=12][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:51.052144] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588C-0-0] [lt=4][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:51.057538] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.057556] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.063463] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=8] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:51.063724] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:51.063731] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:51.063735] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:51.063737] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:51.063740] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:51.063742] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:51.063744] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:51.063746] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:51.063748] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:51.063750] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:51.063751] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:51.063753] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:51.063755] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:51.063757] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:51.063763] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:51.063765] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:51.063767] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:51.063769] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:51.063771] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:51.063773] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:51.063775] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:51.063779] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:51.063783] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:51.063784] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:51.063786] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:51.063790] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:51.063806] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D70-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:51.063809] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D70-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:51.063811] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:51.063813] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:51.063815] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:51.063817] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337651063685, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:51.063820] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:51.063870] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:51.076145] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337650828388]}) [2024-09-26 16:00:51.084998] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:51.085111] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=49, refresh_score_time=27, wash_time=2) [2024-09-26 16:00:51.085730] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=1195366, clean_num=31457) [2024-09-26 16:00:51.086855] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=7] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:51.086881] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337650085699]) [2024-09-26 16:00:51.086890] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337650085699], now0=[mts=1727337650085699], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:51.086896] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337650085295, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:51.086911] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:51.086919] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337650085295, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:51.086926] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337650085295, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:51.086946] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=19][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:51.086951] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:51.086964] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:51.086967] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:51.086972] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:51.086975] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=13, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:51.086979] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=13000, remain_us=15907353, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:51.087825] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:51.087842] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.087852] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:51.099569] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: SHARED_BLOCK_ReaderWriter_IO, mode: read, size: 4096.00, iops: 56.66, rt: 2078.30) [2024-09-26 16:00:51.099578] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:51.099598] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10785.00) [2024-09-26 16:00:51.099601] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:51.099607] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:51.099611] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:51.100198] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:13, local_retry_times:13, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:51.100213] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:51.100218] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:51.100220] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:51.100225] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:51.100263] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:51.100269] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:51.118139] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:51.118150] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.145467] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:51.145482] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337651145463) [2024-09-26 16:00:51.145488] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337651044884, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:51.145506] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223380381947, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:51.145513] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:51.145521] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:51.145527] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:51.145530] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:51.145537] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.145539] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.145541] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337651145492) [2024-09-26 16:00:51.148402] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:00:51.148408] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.175732] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:51.176703] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337650828388]}) [2024-09-26 16:00:51.176723] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D91E-0-0] [lt=8][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337651176697}) [2024-09-26 16:00:51.178675] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.178683] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.207205] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:51.207218] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:51.207223] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:51.209068] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:51.209075] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.235307] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=17] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:51.239226] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.239239] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.245727] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:51.245741] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337651245724) [2024-09-26 16:00:51.245746] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337651145492, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:51.245762] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223380481572, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:51.245768] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:51.245775] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:51.245781] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:51.245784] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:51.245789] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.245791] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.245793] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337651245749) [2024-09-26 16:00:51.247781] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:51.247794] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:51.247798] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:51.247802] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:51.247808] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=5][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:51.247811] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:51.247814] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:51.247817] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:51.247819] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:51.247821] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:51.247823] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:51.247825] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:51.247828] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:51.247830] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:51.247837] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:51.247840] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:51.247845] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:51.247847] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:51.247850] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:51.247853] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:51.247856] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:51.247861] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:51.247867] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:51.247869] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:51.247871] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:51.247877] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:51.247881] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:51.247884] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:51.247889] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:51.247891] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:51.247894] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:51.247896] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337651247711, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:51.247900] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:51.247902] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:51.247941] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=3][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:51.247945] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=4][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:51.247948] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=3][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:51.247950] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:51.247956] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=5][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:51.247959] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:51.247961] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589B-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:51.269276] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.269285] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.277301] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337650828388]}) [2024-09-26 16:00:51.286198] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:51.286299] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:51.286885] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=9, clean_time=0, clean_start_pos=1226823, clean_num=31457) [2024-09-26 16:00:51.299760] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:51.299768] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.310538] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=45][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337652239843, now=[mts=1727337650310107]) [2024-09-26 16:00:51.310546] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337652239843, now=[mts=1727337650310107], now0=[mts=1727337650310107], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:51.310553] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92691614a0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337650309377, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:51.310572] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f9263676038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92691614a0}) [2024-09-26 16:00:51.310580] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92691614a0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337650309377, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:51.310586] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92691614a0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337650309377, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337650310088, use_das=false, nested_level=0, session={this:0x7f9263676038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92691614a0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:51.310603] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:51.310607] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:51.310610] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:51.310612] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:51.310614] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:51.310618] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:51.310621] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=999222, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337652309843) [2024-09-26 16:00:51.310625] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:51.310631] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:51.310633] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:51.310635] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:51.310640] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:51.310673] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:51.310676] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:51.329243] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.329253] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337651329241], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:51.329260] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337651329241], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:51.329264] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=5542, get_gts_with_stc_cnt=12683, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-09-26 16:00:51.330044] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:51.330051] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.345785] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:51.345796] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337651345782) [2024-09-26 16:00:51.345801] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337651245749, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:51.345817] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223380581281, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:51.345823] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:51.345831] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:51.345836] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:51.345841] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:51.345847] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.345849] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.345851] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337651345804) [2024-09-26 16:00:51.360371] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:51.360377] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.376944] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:51.377912] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337651329241]}) [2024-09-26 16:00:51.388473] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=10] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:51.388568] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=17] current monitor number(seq_=-1) [2024-09-26 16:00:51.390961] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:51.390972] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.397763] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D3EA6-0-0] [lt=14][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:51.406231] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337651406230, bytes: 14672632, bw: 0.003616 MB/s, add_ts: 1007979, add_bytes: 3822 [2024-09-26 16:00:51.421856] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:51.421863] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.445816] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] ====== tenant freeze timer task ====== [2024-09-26 16:00:51.445829] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:51.445832] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:51.445842] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337651445829) [2024-09-26 16:00:51.445850] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337651345804, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:51.445860] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337650629877, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337650629842, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337650629895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:51.445871] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223380681765, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:51.445877] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:51.445883] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:51.445888] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:51.445894] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:51.445896] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:51.445914] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.445916] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.445918] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337651445856) [2024-09-26 16:00:51.445942] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DE-0-0] [lt=2] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:51.445952] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DE-0-0] [lt=7] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:51.445957] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DE-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:51.445960] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DE-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:51.445963] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DE-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:51.445966] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DE-0-0] [lt=3][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:51.445969] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DE-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:51.445978] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DE-0-0] [lt=9] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:51.445980] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DE-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:51.445982] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DE-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:51.445985] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DE-0-0] [lt=1][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:51.446023] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851DE-0-0] [lt=4] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:51.451142] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=17][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337651329241]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:51.451153] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=11][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:51.452153] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:51.452158] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.460967] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.460976] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.460981] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.460987] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.460990] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.460995] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.461004] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.461009] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.461013] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.461589] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.461600] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.461605] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.461642] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.461649] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.461653] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.461659] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.461661] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.461663] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.462877] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.462880] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.462883] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.462924] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.462940] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.462945] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.462950] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.462953] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.462955] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.463848] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.463855] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.463859] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.463863] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.463865] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.463864] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.463867] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.463868] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.463872] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.464690] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.464697] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.464700] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.464703] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.464703] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.464706] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.464712] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.464714] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.464715] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.465812] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.465815] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.465817] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:51.465817] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.465822] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.466795] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.466801] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.466800] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.467398] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.467402] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.467407] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.468728] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.468733] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.468740] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.469705] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.469694] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.469794] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.470241] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.470314] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.470325] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=95][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.471657] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.471681] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.471686] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.472283] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.472284] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.472289] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.473602] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.473632] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.473637] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.474578] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.474619] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.474625] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.475446] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.475452] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.475460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.476559] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.476561] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.476565] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.477516] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.477524] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.477529] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.478517] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.478516] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.478524] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.478529] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337651329241]}) [2024-09-26 16:00:51.479478] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.479484] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.479510] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.480445] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.480465] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.480485] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.481419] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.481460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.481472] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.482445] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=36][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.482448] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.482453] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.482453] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.482487] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.483412] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.483429] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.483431] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.484355] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.484362] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.484363] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.484944] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.484946] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.484950] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.486316] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.486323] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.486346] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.486936] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.487378] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:51.487465] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.487530] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=52, refresh_score_time=51, wash_time=2) [2024-09-26 16:00:51.488124] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=1258280, clean_num=31457) [2024-09-26 16:00:51.488161] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.488166] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.488249] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.488692] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.488699] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.489119] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.489213] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.489240] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.490212] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.490232] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.490238] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.490754] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.490770] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.490776] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.491418] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.491489] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.491495] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:51.512514] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:51.512524] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.526540] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591F-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:51.526552] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591F-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:51.526561] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591F-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:51.526566] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718591F-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:51.542987] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:51.542997] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.545874] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:171) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] Eliminate task evict sql audit(request_manager_->get_tenant_id()=1, queue_size=100000, config_mem_limit_=64424509, request_manager_->get_size_used()=6528, evict_high_size_level=90000, evict_low_size_level=80000, allocator->allocated()=12478464, evict_high_mem_level=32212254, evict_low_mem_level=0) [2024-09-26 16:00:51.545904] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=29] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6528, mem_used=12478464) [2024-09-26 16:00:51.545930] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:51.545938] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337651545928) [2024-09-26 16:00:51.545943] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337651445855, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:51.545949] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:51.545958] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:51.545961] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:51.546295] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:51.546306] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:51.546310] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:51.546313] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:51.546316] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:51.546319] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:51.546323] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:51.546325] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:51.546330] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:51.546332] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:51.546334] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:51.546337] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:51.546339] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:51.546341] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:51.546349] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:51.546352] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:51.546355] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:51.546358] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:51.546362] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:51.546366] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:51.546368] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:51.546374] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:51.546380] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=5][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:51.546390] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885765-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:51.546421] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337651546421, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=468, wlock_time=10, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:51.546432] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:51.546449] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223380782419, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:51.546454] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:51.546461] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:51.546466] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:51.546468] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:51.546474] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.546476] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.546500] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337651546436) [2024-09-26 16:00:51.564496] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=11] gc stale ls task succ [2024-09-26 16:00:51.573270] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:51.573277] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.577147] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:51.578637] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337651329241]}) [2024-09-26 16:00:51.597220] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:13, tid:2932}]) [2024-09-26 16:00:51.601575] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.601589] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.602544] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.602545] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.603515] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.603522] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.633819] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.633827] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.646513] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:51.646521] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:51.646530] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337651646518) [2024-09-26 16:00:51.646535] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337651545946, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:51.646550] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223380883018, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:51.646557] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:51.646565] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:51.646570] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:51.646575] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:51.646580] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.646582] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.646584] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337651646539) [2024-09-26 16:00:51.662236] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:51.664029] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.664039] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.679195] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337651329241]}) [2024-09-26 16:00:51.680192] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855ED-0-0] [lt=3] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:51.688349] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:51.688480] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=17] Wash time detail, (compute_wash_size_time=59, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:51.689068] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=1289737, clean_num=31457) [2024-09-26 16:00:51.695376] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:51.695387] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.697305] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4184, dropped:505, tid:2878}]) [2024-09-26 16:00:51.724657] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:51.725605] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:51.725612] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.736349] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=5] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:51.747123] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:51.747139] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337651747120) [2024-09-26 16:00:51.747145] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337651646539, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:51.747163] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223380982730, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:51.747170] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:51.747177] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:51.747193] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:51.747196] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:51.747203] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.747205] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.747207] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337651747148) [2024-09-26 16:00:51.755880] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:51.755886] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.757851] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=8][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337651329241]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:51.757873] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=22][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:51.777423] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:51.780357] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337651329241]}) [2024-09-26 16:00:51.785947] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:51.785954] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.816472] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.816479] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.818427] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [2915][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=10] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-09-26 16:00:51.821374] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.821387] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.821390] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.821394] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.821400] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.821406] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.821414] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.821421] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.821429] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.822297] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.823293] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:51.830231] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:51.830242] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337651830229], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:51.830262] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337651830229], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:51.830267] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:51.830271] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:51.844532] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337651844531, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007741, add_bytes: 0 [2024-09-26 16:00:51.847721] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:51.847731] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337651847718) [2024-09-26 16:00:51.847735] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337651747148, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:51.847753] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223381084162, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:51.847772] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:51.847780] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:51.847785] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:51.847787] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:51.847794] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.847796] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.847798] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337651847739) [2024-09-26 16:00:51.847882] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.847886] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.872253] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337651798517, now=[mts=1727337650870174]) [2024-09-26 16:00:51.872265] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337651798517, now=[mts=1727337650870174], now0=[mts=1727337650870174], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:51.872272] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a311e400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500787, assoc_session_id:3221500787, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337650870009, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:51.872289] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a2319bf8, id:3221500787, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a311e400}) [2024-09-26 16:00:51.872297] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500787, *tx_desc={this:0x7f92a311e400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500787, assoc_session_id:3221500787, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337650870009, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:51.872305] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500787, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a311e400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500787, assoc_session_id:3221500787, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337650870009, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337649868835, use_das=false, nested_level=0, session={this:0x7f92a2319bf8, id:3221500787, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a311e400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:51.872322] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:51.872327] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:51.872330] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:51.872333] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:51.872336] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:51.872340] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:51.872345] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:51.872352] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:51.872355] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:51.872356] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:51.872362] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:51.872365] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2003530) [2024-09-26 16:00:51.872368] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885087-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:51.872372] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:51.872375] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:51.872380] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:51.872383] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8060, start=1727337649868830, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:51.872386] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:00:51.872389] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:51.872421] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:51.872424] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:51.872427] WDIAG [SHARE] get_snapshot_gc_scn (ob_global_stat_proxy.cpp:164) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:51.872431] WDIAG [STORAGE] get_global_info (ob_tenant_freeze_info_mgr.cpp:811) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] fail to get global info(ret=-4012, tenant_id=1) [2024-09-26 16:00:51.872434] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:954) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] failed to get global info(ret=-4012) [2024-09-26 16:00:51.872436] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1008) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to try update info(tmp_ret=-4012, tmp_ret="OB_TIMEOUT") [2024-09-26 16:00:51.872441] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1044) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2] success to update min reserved snapshot(reserved_snapshot=0, duration=1800, snapshot_gc_ts_=0) [2024-09-26 16:00:51.872445] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1071) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3] update reserved snapshot finished(cost_ts=4, reserved_snapshot=0) [2024-09-26 16:00:51.877979] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:51.877986] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.880932] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337651830229]}) [2024-09-26 16:00:51.889884] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:51.889999] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash time detail, (compute_wash_size_time=127, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:51.890642] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=1321194, clean_num=31457) [2024-09-26 16:00:51.908267] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.908274] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.938590] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.938598] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.949324] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:51.949335] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337651949321) [2024-09-26 16:00:51.949340] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337651847738, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:51.949356] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223381185785, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:51.949362] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:51.949379] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:51.949384] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:51.949386] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:51.949392] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.949394] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:51.949396] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337651949343) [2024-09-26 16:00:51.968812] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:51.968818] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:51.977609] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:51.981519] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337651830229]}) [2024-09-26 16:00:51.983484] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] tx gc loop thread is running(MTL_ID()=1) [2024-09-26 16:00:51.983492] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] try gc retain ctx [2024-09-26 16:00:51.983876] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-09-26 16:00:51.983900] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:51.999081] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:51.999088] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.020853] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185920-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:52.020864] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185920-0-0] [lt=11][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:52.020883] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185920-0-0] [lt=18][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:52.020889] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185920-0-0] [lt=6][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:52.026620] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500796, support_ssl=false) [2024-09-26 16:00:52.026651] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221500796, sess.client_addr_="127.0.0.1:36364") [2024-09-26 16:00:52.026656] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f92be35cf90, session_id:3221500796, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:52.026677] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:52.026685] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=2] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:52.026721] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA2-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:52.026898] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA2-0-0] [lt=2][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:52.026905] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA2-0-0] [lt=6][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"lplO%h_&90"Z!1qj>8;a"}, ret=-4043) [2024-09-26 16:00:52.026911] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA2-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:52.026915] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA2-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:52.026922] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA2-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:52.026944] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA2-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500796, proxy_sessid:0}) [2024-09-26 16:00:52.026948] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA2-0-0] [lt=4] mark session id unused(sessid=3221500796) [2024-09-26 16:00:52.026951] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA2-0-0] [lt=2][errcode=0] server close connection(sessid=3221500796, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:52.026955] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA2-0-0] [lt=4][errcode=-4018] get session fail(ret=-4018, sessid=3221500796, proxy_sessid=0) [2024-09-26 16:00:52.026957] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA2-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:52.026960] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA2-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500796, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:52.026997] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f92be35cf90, session_id:3221500796, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337652026624, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:52.027007] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] kill and revert session(conn.sessid_=3221500796, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:52.027010] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be35cf90, session_id:3221500796, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337652026624, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:52.027014] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500796, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:52.029386] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:52.029394] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.049919] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:52.049926] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:52.049933] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337652049916) [2024-09-26 16:00:52.049937] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337651949343, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:52.049942] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:52.049951] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:52.049953] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:52.050264] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:52.050272] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:52.050275] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:52.050279] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:52.050282] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:52.050284] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:52.050287] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:52.050290] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:52.050291] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:52.050295] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=3][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:52.050297] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:52.050299] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:52.050302] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:52.050304] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:52.050310] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:52.050312] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:52.050314] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:52.050317] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:52.050319] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:52.050321] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:52.050324] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:52.050330] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:52.050335] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:52.050338] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:52.050339] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:52.050344] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:52.050347] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:52.050350] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885766-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:52.050352] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:52.050354] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:52.050357] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:52.050360] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337652050187, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:52.050364] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:52.050366] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:52.050392] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:52.050395] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337652050394, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=447, wlock_time=8, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:52.050400] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:52.050403] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:52.050418] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223381286409, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:52.050424] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:52.050429] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:52.050433] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:52.050435] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:52.050440] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.050442] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.050444] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337652050406) [2024-09-26 16:00:52.054067] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:52.054079] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:52.054083] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:52.054100] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:52.054104] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:52.054106] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:52.054112] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=5][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:52.054114] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:52.054117] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:52.054118] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:52.054120] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:52.054121] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:52.054123] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:52.054126] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:52.054131] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:52.054134] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:52.054137] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:52.054140] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:52.054144] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:52.054147] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:52.054149] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:52.054154] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:52.054158] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:52.054160] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:52.054162] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:52.054166] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:52.054169] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588D-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:52.054174] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588D-0-0] [lt=4][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:52.054177] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588D-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:52.054179] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:52.054181] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:52.054184] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588D-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337652054015, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:52.054187] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588D-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:52.054189] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588D-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:52.054218] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588D-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:52.054221] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8588D-0-0] [lt=3][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:52.059656] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:52.059663] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.064103] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=6] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:52.064347] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:52.064356] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:52.064359] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:52.064384] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=25][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:52.064416] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=22][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:52.064419] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:52.064422] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:52.064425] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:52.064427] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:52.064429] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:52.064431] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:52.064433] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:52.064435] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:52.064437] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:52.064441] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:52.064444] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:52.064446] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:52.064449] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:52.064453] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:52.064474] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=20][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:52.064476] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:52.064481] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:52.064486] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:52.064488] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:52.064490] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:52.064495] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:52.064498] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D72-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:52.064503] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D72-0-0] [lt=5][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:52.064506] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:52.064508] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:52.064510] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:52.064512] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337652064310, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:52.064516] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:52.064561] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:52.082119] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=28] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337651830229]}) [2024-09-26 16:00:52.087049] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=6] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:52.089959] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:52.089970] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.089979] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:52.092050] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:52.092164] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Wash time detail, (compute_wash_size_time=59, refresh_score_time=16, wash_time=2) [2024-09-26 16:00:52.092798] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=1352651, clean_num=31457) [2024-09-26 16:00:52.100703] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=23][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337651100262]) [2024-09-26 16:00:52.100713] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337651100262], now0=[mts=1727337651100262], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:52.100720] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337651099551, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:52.100736] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:52.100744] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337651099551, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:52.100752] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337651099551, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:52.100768] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:52.100773] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:52.100777] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:52.100780] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:52.100782] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:52.100786] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=14, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:52.100790] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=14000, remain_us=14893542, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=14, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:52.104619] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 12111.00) [2024-09-26 16:00:52.104627] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:52.104639] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:52.104642] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:52.114995] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:14, local_retry_times:14, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:52.115010] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:52.115013] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:52.115015] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:52.115021] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:52.115059] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:52.115062] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:52.120069] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:52.120080] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.150491] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:52.150503] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.150543] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:52.150552] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337652150540) [2024-09-26 16:00:52.150558] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337652049939, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:52.150577] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223381386959, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:52.150583] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:52.150592] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:52.150598] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:52.150601] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:52.150606] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.150609] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.150611] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337652150561) [2024-09-26 16:00:52.177779] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:52.180763] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:00:52.180771] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.182714] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337651830229]}) [2024-09-26 16:00:52.182741] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D92A-0-0] [lt=7][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337652182708}) [2024-09-26 16:00:52.200328] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=2] ====== check clog disk timer task ====== [2024-09-26 16:00:52.200335] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=5] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:52.200344] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:52.211073] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:52.211080] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.241329] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:52.241335] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.248510] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:52.248521] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:52.248526] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:52.248529] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:52.248533] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:52.248535] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:52.248539] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:52.248541] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:52.248543] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:52.248546] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:52.248548] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:52.248550] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:52.248553] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:52.248555] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:52.248561] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:52.248564] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:52.248566] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:52.248569] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:52.248571] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:52.248573] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:52.248576] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:52.248581] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:52.248586] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:52.248588] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:52.248590] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:52.248595] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:52.248598] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:52.248601] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:52.248603] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:52.248606] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:52.248608] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:52.248610] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337652248439, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:52.248613] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:52.248615] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:52.248665] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=3][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:52.248669] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=4][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:52.248671] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:52.248674] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:52.248676] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:52.248679] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:52.248681] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589C-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:52.250653] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:52.250667] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337652250650) [2024-09-26 16:00:52.250673] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337652150561, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:52.250690] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223381486688, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:52.250696] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:52.250704] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:52.250709] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:52.250712] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:52.250717] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.250720] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.250722] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337652250676) [2024-09-26 16:00:52.271605] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:52.271612] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.283310] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337651830229]}) [2024-09-26 16:00:52.293182] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:52.293286] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=53, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:52.293960] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=5, clean_time=0, clean_start_pos=1384108, clean_num=31457) [2024-09-26 16:00:52.301872] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:52.301878] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.311630] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337652239843, now=[mts=1727337651310672]) [2024-09-26 16:00:52.311639] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337652239843, now=[mts=1727337651310672], now0=[mts=1727337651310672], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:52.311644] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92691614a0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337651310522, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:52.311661] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f9263676038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92691614a0}) [2024-09-26 16:00:52.311668] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=6][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92691614a0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337651310522, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:52.311675] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92691614a0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337651310522, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337650310088, use_das=false, nested_level=0, session={this:0x7f9263676038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92691614a0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:52.311691] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:52.311695] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:52.311698] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:52.311700] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:52.311702] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:52.311705] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:52.311709] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:52.311715] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:52.311717] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=1][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:52.311718] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:52.311723] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:52.311726] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2001637) [2024-09-26 16:00:52.311728] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:52.311731] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:52.311733] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:52.311735] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:52.311738] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:52.311742] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:52.311768] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:52.311770] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:52.311772] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:52.311775] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:52.311778] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:52.311780] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:52.311886] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C80-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:52.311892] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:52.311894] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:52.311909] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:52.311914] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:52.312189] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:52.312197] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:52.331304] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.331314] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337652331302], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:52.331321] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337652331302], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:52.332135] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:52.332143] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.350752] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:52.350771] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337652350750) [2024-09-26 16:00:52.350787] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337652250676, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:52.350803] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223381587206, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:52.350811] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:52.350818] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:52.350823] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:52.350825] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:52.350831] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.350833] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.350835] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337652350791) [2024-09-26 16:00:52.362440] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:52.362447] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.378064] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:52.383925] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337652331302]}) [2024-09-26 16:00:52.388794] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=18] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:52.392697] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:52.392717] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.399560] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D3F1C-0-0] [lt=9][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:52.408087] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=1] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:52.408102] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:52.408110] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:52.408114] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:52.408120] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=5][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:52.408123] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:52.408125] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:52.408127] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:52.408131] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:52.408134] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:52.408136] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:52.408138] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:52.408140] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:52.408143] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:52.408146] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:52.408148] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:52.408156] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:52.408158] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:52.408161] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:52.408164] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:52.408167] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-09-26 16:00:52.408173] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=2][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:52.408178] WDIAG [SERVER] do_process (obmp_query.cpp:771) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=5][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-09-26 16:00:52.408184] WDIAG [SERVER] do_process (obmp_query.cpp:903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, session={this:0x7f92a230bdf8, id:3221487638, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-09-26 16:00:52.408195] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F51-0-0] [lt=7] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:52.413283] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337652413281, bytes: 14676172, bw: 0.003352 MB/s, add_ts: 1007051, add_bytes: 3540 [2024-09-26 16:00:52.423160] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:52.423172] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.451320] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:52.451336] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337652451316) [2024-09-26 16:00:52.451342] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337652350791, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:52.451362] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223381686850, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:52.451369] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:52.451378] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:52.451383] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:52.451386] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:52.451392] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.451395] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.451397] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337652451346) [2024-09-26 16:00:52.453289] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:52.453297] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.461082] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.461098] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.461114] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=16][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.462067] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.462077] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.462083] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.462106] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.462110] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.462113] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.463993] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.463996] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.463998] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.463998] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.464005] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.464010] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.465971] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.465982] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.465987] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.466014] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.466017] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.466019] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.466929] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.466935] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.466938] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.466958] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.466963] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.466967] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.467939] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.467944] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.467947] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.467949] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.467952] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.467955] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.469887] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.469863] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.469893] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.469892] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=28][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.469895] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.469896] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.470854] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.470859] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.470862] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.470915] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.470947] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=31][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.470950] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.471382] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.471388] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.471391] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:52.471473] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.472822] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.472838] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.473757] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.473766] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.474752] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.474757] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.475731] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.475730] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.477698] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.477701] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.478661] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.478662] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.479632] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.479643] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.480615] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.480623] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.481578] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.481590] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.482596] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.482597] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.483565] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:52.483569] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.483576] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.483573] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.484517] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.484529] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337652331302]}) [2024-09-26 16:00:52.484543] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.485502] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.485532] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.486458] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.486490] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.487425] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.487461] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.488441] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.488462] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.489407] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.489411] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.490354] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.490365] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.491377] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.491379] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.492326] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.492348] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.493308] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.494336] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.494336] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.494422] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:52.494538] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:52.494918] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.494955] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.495149] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1415565, clean_num=31457) [2024-09-26 16:00:52.496235] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.496251] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.497205] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.497217] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.498231] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.498232] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.499172] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.499173] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.500141] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.500148] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.501103] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.501126] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.502118] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.502118] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.503080] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.503080] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.504054] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.504058] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.505073] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.505085] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.506011] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.506976] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.506990] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.507954] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.508921] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.508924] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.509896] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.509929] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.510864] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.510891] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.511850] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.512828] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.512858] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.513791] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.513810] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:52.513818] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.513815] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.514778] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.515804] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.515828] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:52.526684] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185921-0-0] [lt=28][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:52.526696] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185921-0-0] [lt=11][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:52.526704] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185921-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:52.526709] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185921-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:52.544092] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:52.544101] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.546053] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6537, mem_used=12478464) [2024-09-26 16:00:52.551904] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:52.551917] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337652551901) [2024-09-26 16:00:52.551923] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337652451345, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:52.551929] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:52.551938] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:52.551942] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:52.552256] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:52.552268] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:52.552272] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:52.552275] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:52.552279] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:52.552281] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:52.552284] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:52.552286] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:52.552289] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:52.552291] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:52.552293] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:52.552295] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:52.552297] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:52.552299] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:52.552320] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:52.552323] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:52.552327] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:52.552329] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:52.552331] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:52.552334] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:52.552336] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:52.552343] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:52.552363] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:52.552367] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:52.552368] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:52.552375] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:52.552378] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:52.552381] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885767-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:52.552384] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:52.552387] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:52.552390] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:52.552414] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337652552182, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:52.552418] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:52.552423] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:52.552454] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:52.552473] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337652552473, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=539, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:52.552481] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:52.552484] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:52.552501] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223381788797, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:52.552506] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:52.552513] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:52.552517] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:52.552519] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:52.552525] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.552528] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.552529] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337652552488) [2024-09-26 16:00:52.564613] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=6] gc stale ls task succ [2024-09-26 16:00:52.574376] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:52.574394] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.578314] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:52.585148] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337652331302]}) [2024-09-26 16:00:52.601714] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.601714] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.601726] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.601771] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.604686] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:52.604714] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=26][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.634959] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:52.634988] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.649909] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD5-0-0] [lt=8] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:52.649923] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD5-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:52.649951] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD5-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:52.652504] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:52.652516] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337652652501) [2024-09-26 16:00:52.652522] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337652551926, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:52.652538] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223381887570, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:52.652544] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:52.652552] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:52.652557] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:52.652559] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:52.652564] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.652568] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.652569] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337652652525) [2024-09-26 16:00:52.652575] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:52.662274] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:52.666240] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:52.666250] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.685718] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337652331302]}) [2024-09-26 16:00:52.695589] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:52.695722] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Wash time detail, (compute_wash_size_time=71, refresh_score_time=17, wash_time=2) [2024-09-26 16:00:52.696326] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=1447022, clean_num=31457) [2024-09-26 16:00:52.697426] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:52.697457] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.728719] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:52.728725] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.729682] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:52.753164] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:52.753180] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337652753160) [2024-09-26 16:00:52.753185] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337652652524, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:52.753211] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223381989642, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:52.753218] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:52.753225] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:52.753230] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:52.753233] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:52.753238] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.753240] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.753242] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337652753198) [2024-09-26 16:00:52.759001] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:52.759010] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.778493] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:52.787317] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337652331302]}) [2024-09-26 16:00:52.788271] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=5][errcode=-4018] try free send task failed(ret=-4018) [2024-09-26 16:00:52.789293] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:52.789297] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.819538] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:52.819546] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.821480] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.821491] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.821500] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.821505] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.821511] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.821515] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.822455] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.822466] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.822468] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.822474] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.822479] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:52.832333] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:52.832343] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337652832331], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:52.832358] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337652832331], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:52.832364] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:52.832368] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:52.849811] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:52.849827] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.852741] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=7] PNIO [ratelimit] time: 1727337652852740, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008209, add_bytes: 0 [2024-09-26 16:00:52.853726] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:52.853742] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337652853723) [2024-09-26 16:00:52.853748] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337652753198, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:52.853764] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223382090226, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:52.853770] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:52.853778] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:52.853783] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:52.853786] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:52.853791] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.853794] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.853796] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337652853752) [2024-09-26 16:00:52.880117] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16] success to notify_need_writing_throttling True [2024-09-26 16:00:52.880124] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.887890] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337652832331]}) [2024-09-26 16:00:52.888868] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:262) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=18] ObArchiveSender is running(thread_index=0) [2024-09-26 16:00:52.896820] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:52.896941] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:52.897548] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=1478479, clean_num=31457) [2024-09-26 16:00:52.910410] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:52.910417] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.940635] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:52.940642] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.954368] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:52.954381] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337652954365) [2024-09-26 16:00:52.954387] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337652853752, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:52.954402] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223382190837, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:52.954409] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:52.954416] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:52.954421] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:52.954424] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:52.954430] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.954432] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:52.954434] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337652954390) [2024-09-26 16:00:52.970911] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:52.970956] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=43][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:52.978786] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:52.984564] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:52.988504] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337652832331]}) [2024-09-26 16:00:53.002189] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:53.002197] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.020852] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185922-0-0] [lt=37][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:53.020863] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185922-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:53.020872] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185922-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:53.020877] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185922-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:53.024544] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA3-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:53.024558] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA3-0-0] [lt=12][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:53.024609] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA3-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:53.024643] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=0] ls blacklist refresh finish(cost_time=429) [2024-09-26 16:00:53.027773] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500800, support_ssl=false) [2024-09-26 16:00:53.027805] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221500800, sess.client_addr_="127.0.0.1:36366") [2024-09-26 16:00:53.027813] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] accept one succ(*s={this:0x7f9264e04030, session_id:3221500800, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:53.027850] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:53.027859] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=5] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:53.027905] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA3-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:53.028073] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA3-0-0] [lt=5][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:53.028080] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA3-0-0] [lt=6][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"c4vws\Re&+EZtZD(7{(S"}, ret=-4043) [2024-09-26 16:00:53.028086] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA3-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:53.028090] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA3-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:53.028099] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA3-0-0] [lt=4] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:53.028118] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA3-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500800, proxy_sessid:0}) [2024-09-26 16:00:53.028124] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA3-0-0] [lt=4] mark session id unused(sessid=3221500800) [2024-09-26 16:00:53.028128] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA3-0-0] [lt=2][errcode=0] server close connection(sessid=3221500800, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:53.028131] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA3-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500800, proxy_sessid=0) [2024-09-26 16:00:53.028133] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA3-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:53.028136] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA3-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500800, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:53.028172] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4015] close sql sock by user req(*s={this:0x7f9264e04030, session_id:3221500800, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337653027683, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:53.028179] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500800, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:53.028183] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9264e04030, session_id:3221500800, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337653027683, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:53.028187] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500800, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:53.032465] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:00:53.032470] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.055194] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588E-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:53.055206] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588E-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:53.055238] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588E-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:53.055272] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:53.055279] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:53.055286] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337653055270) [2024-09-26 16:00:53.055291] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337652954390, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:53.055296] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:53.055304] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:53.055307] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:53.055559] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885768-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:53.055564] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885768-0-0] [lt=5][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:53.055589] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885768-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:53.055610] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337653055610, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=309, wlock_time=7, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:53.055618] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:53.055634] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223382290398, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:53.055639] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:53.055645] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:53.055650] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:53.055652] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:53.055658] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.055660] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.055662] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337653055622) [2024-09-26 16:00:53.062753] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:53.062762] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.064706] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=10] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:53.064959] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D74-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:53.064969] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D74-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:53.064987] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D74-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:53.087168] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=8] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:53.089103] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337652832331]}) [2024-09-26 16:00:53.093023] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:53.093029] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.093037] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:53.097992] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:53.098093] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:53.098713] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=1509936, clean_num=31457) [2024-09-26 16:00:53.104741] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:176, tid:2932}]) [2024-09-26 16:00:53.110612] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10702.00) [2024-09-26 16:00:53.110627] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:53.110639] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:53.110643] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:53.115497] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337652115058]) [2024-09-26 16:00:53.115504] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337652115058], now0=[mts=1727337652115058], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:53.115511] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337652114339, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:53.115527] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:53.115539] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337652114339, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:53.115546] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337652114339, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:53.115564] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:53.115569] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:53.115572] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:53.115574] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:53.115578] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:53.115582] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=15, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:53.115586] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=15000, remain_us=13878746, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:53.124379] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:53.124386] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.132086] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:15, local_retry_times:15, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:53.132098] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:53.132101] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:53.132103] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:53.132108] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:53.132146] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:53.132148] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:53.145693] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:221) [2596][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=9] [ASYNC TASK QUEUE](queue_size=0) [2024-09-26 16:00:53.154500] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:53.154511] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.155520] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:53.155536] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337653155515) [2024-09-26 16:00:53.155542] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337653055294, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:53.155563] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223382391951, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:53.155570] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:53.155578] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:53.155584] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:53.155587] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:53.155605] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.155608] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.155610] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337653155546) [2024-09-26 16:00:53.179230] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=817856, replace_num=15728) [2024-09-26 16:00:53.179242] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:53.185788] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:53.185826] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.189740] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337652832331]}) [2024-09-26 16:00:53.189759] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D936-0-0] [lt=8][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337653189732}) [2024-09-26 16:00:53.207543] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:53.207553] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:53.207560] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:53.216046] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:53.216056] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.246377] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=37] success to notify_need_writing_throttling True [2024-09-26 16:00:53.246396] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.249695] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:53.249705] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:53.249709] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:53.249713] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:53.249716] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:53.249719] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:53.249722] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:53.249724] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:53.249726] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:53.249728] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:53.249730] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:53.249732] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:53.249735] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:53.249737] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:53.249743] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:53.249745] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:53.249748] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:53.249750] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:53.249752] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:53.249755] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:53.249758] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:53.249763] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:53.249768] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:53.249770] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:53.249772] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:53.249777] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:53.249780] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:53.249783] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:53.249785] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:53.249787] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:53.249789] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:53.249792] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337653249625, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:53.249795] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:53.249797] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:53.249825] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=3][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:53.249828] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:53.249830] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:53.249832] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=1][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:53.249834] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=1][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:53.249837] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:53.249838] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589D-0-0] [lt=1][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:53.255558] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:53.255587] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337653255556) [2024-09-26 16:00:53.255594] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337653155546, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:53.255611] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223382491603, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:53.255619] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:53.255627] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:53.255633] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:53.255636] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:53.255644] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.255647] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.255649] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337653255598) [2024-09-26 16:00:53.276635] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:53.276645] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.283517] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=28] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:53.290304] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337652832331]}) [2024-09-26 16:00:53.299209] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:53.299312] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=53, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:53.299947] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=16] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=1541393, clean_num=31457) [2024-09-26 16:00:53.306917] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:53.306927] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.311789] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=31][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337654241919, now=[mts=1727337652312188]) [2024-09-26 16:00:53.311798] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337654241919, now=[mts=1727337652312188], now0=[mts=1727337652312188], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:53.311804] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269163240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337652311625, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:53.311821] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269163240}) [2024-09-26 16:00:53.311829] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269163240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337652311625, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:53.311838] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=8] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269163240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337652311625, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337652312160, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269163240}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:53.311854] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:53.311859] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:53.311862] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:53.311865] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:53.311867] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:53.311871] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:53.311877] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=1000043, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337654311919) [2024-09-26 16:00:53.311881] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:53.311888] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:53.311890] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:53.311892] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:53.311897] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:53.311933] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:53.311938] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:53.318489] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD6-0-0] [lt=1][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-09-26 16:00:53.318497] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD6-0-0] [lt=8][errcode=-4002] refresh sever list failed(ret=-4002) [2024-09-26 16:00:53.318499] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD6-0-0] [lt=2][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-09-26 16:00:53.333390] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.333399] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337653333387], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:53.333405] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337653333387], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:53.333408] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337653333387]}) [2024-09-26 16:00:53.337039] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:53.337048] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.355595] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223382591253, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:53.355606] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:53.355614] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:53.355619] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:53.355622] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:53.355628] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.355630] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.355632] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337653355581) [2024-09-26 16:00:53.367267] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:53.367275] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.379302] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:53.388934] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=26] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:53.388946] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=5] current monitor number(seq_=-1) [2024-09-26 16:00:53.390907] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337653333387]}) [2024-09-26 16:00:53.397763] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:53.397769] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.400672] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D3F64-0-0] [lt=14][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:53.420979] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1727337653420979, bytes: 14679994, bw: 0.003617 MB/s, add_ts: 1007698, add_bytes: 3822 [2024-09-26 16:00:53.427814] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:53.427824] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.434856] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:218) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-09-26 16:00:53.434867] INFO [COMMON] handle (memory_dump.cpp:498) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-09-26 16:00:53.434894] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] update_check_range(min_check_version=342, max_check_version=343, global_version=344) [2024-09-26 16:00:53.437446] INFO handle (memory_dump.cpp:626) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] statistics: tenant_cnt: 3, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 80 198 0 1 2 362 1 0 1 4 1 1 0 1 5 1 5 0 1 7 1 2 0 1 8 60 1 0 1 12 4 2 0 1 13 2 1 0 1 14 1 1 0 1 18 54 3 0 500 0 52 216 0 500 7 2 3 0 500 8 42 1 0 500 9 2 1 0 500 10 6 2 0 500 16 1 1 0 500 17 6 7 0 500 22 3 49 0 500 23 23 10 0 508 0 2 8 0 508 8 3 1 0 cost_time: 2555 [2024-09-26 16:00:53.456338] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:53.456354] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337653456334) [2024-09-26 16:00:53.456371] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337653255597, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:53.456399] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223382691895, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:53.456409] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:53.456417] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:53.456423] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:53.456426] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:53.456433] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.456435] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.456437] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337653456375) [2024-09-26 16:00:53.456434] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] ====== tenant freeze timer task ====== [2024-09-26 16:00:53.456444] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:53.456488] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337650629877, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337650629842, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337650629895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:53.456511] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:53.456553] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DF-0-0] [lt=5] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:53.456565] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DF-0-0] [lt=10] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:53.456571] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DF-0-0] [lt=4] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:53.456576] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DF-0-0] [lt=4] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:53.456581] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DF-0-0] [lt=5][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:53.456585] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DF-0-0] [lt=3][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:53.456588] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DF-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:53.456598] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DF-0-0] [lt=10] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:53.456600] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DF-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:53.456603] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DF-0-0] [lt=3] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:53.456607] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851DF-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:53.456638] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851DF-0-0] [lt=3] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:53.458290] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:53.458307] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.461746] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.461754] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.461760] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.461778] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.461783] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.461788] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.462602] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.462610] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.462614] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.462693] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.462698] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.462701] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.463170] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.463175] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.463177] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.464159] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.464169] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.464173] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.464171] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.464177] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.464176] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.465115] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.465119] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.465121] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.465124] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.465129] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.465133] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.466094] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.466099] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.466101] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.466156] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.466163] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.466166] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.467085] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.467090] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.467092] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.467098] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.467110] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.467113] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.468031] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.468037] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.468040] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.468048] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.468052] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.468054] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.469022] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.469023] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.469026] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:53.469028] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.470027] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.470039] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.471983] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.471988] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.472994] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.473039] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.473911] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.473924] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.474905] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.474920] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.475442] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.475492] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.476832] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.476856] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.477823] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.477821] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.478381] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.478385] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.478902] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.478942] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.479444] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.479670] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.480210] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.480783] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.481370] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.481409] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.481988] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.482115] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.482711] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.482934] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.483640] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.483892] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.484579] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.484750] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.485278] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.485461] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.486463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.486489] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.487198] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.487497] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.488189] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.488204] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.488402] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:53.488410] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.488788] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.489051] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.489355] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.489895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.490070] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.490471] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.490651] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.491479] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.491485] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.491498] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337653333387]}) [2024-09-26 16:00:53.492464] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.492464] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.493014] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.493015] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.493686] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.494465] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.494466] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.495080] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.495656] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.495750] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.496233] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.496828] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.497122] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.497754] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.497789] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.498523] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.498872] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.499094] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.499671] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.499707] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.500134] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:53.500284] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash time detail, (compute_wash_size_time=66, refresh_score_time=16, wash_time=2) [2024-09-26 16:00:53.500281] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.500296] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1572850, clean_num=31457) [2024-09-26 16:00:53.500311] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.500908] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.500914] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.501585] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.501617] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.502161] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.502162] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.502752] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.502795] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.503533] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.503533] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:53.518850] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:53.518859] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.526783] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185923-0-0] [lt=34][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:53.526796] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185923-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:53.526806] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185923-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:53.526812] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185923-0-0] [lt=6][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:53.535487] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] LogLoopThread round_cost_time(us)(round_cost_time=9) [2024-09-26 16:00:53.546153] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6546, mem_used=12478464) [2024-09-26 16:00:53.549153] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:53.549164] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.556930] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:53.556940] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337653556927) [2024-09-26 16:00:53.556945] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337653456375, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:53.556950] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:53.556960] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:53.556963] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:53.557299] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:53.557312] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:53.557316] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:53.557320] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:53.557323] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:53.557325] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:53.557328] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:53.557331] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:53.557333] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:53.557335] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:53.557336] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:53.557339] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:53.557341] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:53.557343] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:53.557362] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:53.557365] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:53.557368] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:53.557370] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:53.557373] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:53.557376] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:53.557381] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:53.557387] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:53.557393] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:53.557396] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:53.557397] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:53.557404] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:53.557407] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:53.557410] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885769-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:53.557413] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:53.557416] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:53.557418] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:53.557421] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337653557223, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:53.557425] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:53.557427] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:53.557456] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:53.557478] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337653557477, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=522, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:53.557486] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:53.557490] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:53.557507] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223382793447, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:53.557512] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:53.557518] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:53.557523] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:53.557536] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:53.557544] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.557546] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.557548] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337653557494) [2024-09-26 16:00:53.564722] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=9] gc stale ls task succ [2024-09-26 16:00:53.579405] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:53.579404] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:53.579415] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.592103] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337653333387]}) [2024-09-26 16:00:53.601854] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.601867] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.601926] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.602062] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.609697] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:53.609706] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.639973] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:53.639980] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.655927] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:53.655937] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:53.655941] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:53.655945] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:53.655948] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:53.655951] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:53.655954] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:53.655957] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:53.655960] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:53.655962] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:53.655964] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:53.655966] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:53.655969] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:53.655971] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:53.655976] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:53.655979] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:53.655981] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:53.655983] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:53.655986] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, ret=-5019) [2024-09-26 16:00:53.655990] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:53.655992] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:53.655997] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:53.656002] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:53.656005] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:53.656006] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:53.656012] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:53.656015] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:53.656017] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD6-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-09-26 16:00:53.656021] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:53.656024] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:53.656027] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:53.656029] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337653655858, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:53.656032] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:53.656034] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f92b4056738, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, columns_str="zone") [2024-09-26 16:00:53.656039] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:53.656067] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:634) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f92b4056738, where_condition="where svr_ip='192.168.0.45' and svr_port=2882", zone_name_holder=) [2024-09-26 16:00:53.656071] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:567) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:53.656074] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:576) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:53.656076] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:53.656082] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:53.657561] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:53.657582] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337653657559) [2024-09-26 16:00:53.657588] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337653556948, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:53.657618] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223382894057, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:53.657626] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:53.657634] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:53.657640] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:53.657643] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:53.657649] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.657651] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.657654] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337653657593) [2024-09-26 16:00:53.662327] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:53.670234] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:53.670240] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.692695] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337653333387]}) [2024-09-26 16:00:53.700515] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:53.700520] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.700624] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:53.700728] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=53, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:53.701353] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=0, clean_num=31457) [2024-09-26 16:00:53.730819] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:53.730825] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.735646] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:53.757776] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:53.757791] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337653757773) [2024-09-26 16:00:53.757796] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337653657593, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:53.757812] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223382993755, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:53.757819] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:53.757827] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:53.757832] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:53.757834] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:53.757839] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.757850] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.757855] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337653757799) [2024-09-26 16:00:53.761056] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:53.761062] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.779436] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:53.791338] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:53.791345] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.793321] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337653333387]}) [2024-09-26 16:00:53.821630] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=8] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.821633] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:53.821649] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.821650] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.821658] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.821689] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.821705] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.821710] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.821714] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.821721] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.822391] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.822395] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.822405] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:53.834441] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:53.834451] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337653834439], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:53.834471] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337653834439], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:53.834476] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:53.834480] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:53.851987] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:53.851994] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.857808] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223383094252, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:53.857830] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:53.857837] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:53.857845] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:53.857848] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:53.857864] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.857868] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.857870] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337653857793) [2024-09-26 16:00:53.859478] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1727337653859476, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006736, add_bytes: 0 [2024-09-26 16:00:53.882326] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:53.882333] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.893674] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=96] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337653834439]}) [2024-09-26 16:00:53.901819] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:53.901931] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash time detail, (compute_wash_size_time=61, refresh_score_time=16, wash_time=2) [2024-09-26 16:00:53.902558] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=31457, clean_num=31457) [2024-09-26 16:00:53.912467] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:53.912473] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.943716] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:53.943724] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.943737] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] BlockGCTimerTask success(ret=0, cost_time_us=23, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.958359] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:53.958371] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337653958355) [2024-09-26 16:00:53.958377] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337653757799, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:53.958410] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=223383194309, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:53.958418] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:53.958424] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:53.958429] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:53.958431] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:53.958437] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.958439] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:53.958441] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337653958390) [2024-09-26 16:00:53.961760] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-09-26 16:00:53.961774] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-09-26 16:00:53.961780] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-09-26 16:00:53.961784] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-09-26 16:00:53.961789] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=4][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:53.961792] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:53.961796] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-09-26 16:00:53.961804] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=7][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:53.961808] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:53.961811] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:53.961813] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:53.961817] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:53.961820] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:53.961823] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:53.961830] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:53.961833] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:53.961838] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:53.961841] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:53.961844] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:53.961847] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:53.961850] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:53.961857] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:53.961864] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:53.961867] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:53.961870] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:53.961878] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:53.961882] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:53.961885] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2024-09-26 16:00:53.961889] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:53.961892] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:53.961895] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:53.961898] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337653961675, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:53.961903] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:53.961906] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1111) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:53.961935] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:801) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:53.961939] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:67) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:53.961942] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:436) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:53.961945] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:95) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:53.961948] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:109) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=2] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:53.962322] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-09-26 16:00:53.962330] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2024-09-26 16:00:53.962336] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2024-09-26 16:00:53.962340] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2024-09-26 16:00:53.962344] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:53.962348] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:53.962352] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_tenant' doesn't exist [2024-09-26 16:00:53.962355] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:53.962358] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:53.962360] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:53.962367] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:53.962370] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:53.962373] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:53.962375] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:53.962379] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:53.962382] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:53.962385] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:53.962388] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:53.962391] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:53.962394] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:53.962397] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:53.962403] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:53.962408] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:53.962410] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:53.962416] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=5][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:53.962421] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:53.962428] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD7-0-0] [lt=7][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:53.962432] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=4][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, aret=-5019, ret=-5019) [2024-09-26 16:00:53.962436] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:00:53.962438] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:53.962441] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:53.962443] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337653962286, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:00:53.962447] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:53.962450] WDIAG [SHARE] read_tenants (ob_unit_table_operator.cpp:1174) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] execute sql failed(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:53.962466] WDIAG [SHARE] get_tenants (ob_unit_table_operator.cpp:108) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] fail read tenants(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:53.962470] WDIAG [SHARE] get_tenants (ob_unit_getter.cpp:198) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=3][errcode=-5019] ut_operator get_resource_pools failed(ret=-5019) [2024-09-26 16:00:53.962473] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:125) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=2][errcode=-5019] get cluster tenants fail(ret=-5019) [2024-09-26 16:00:53.962485] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:325) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=2] cal_all_part_disk_default_percentage succ(data_dir="/root/observer/store/sstable", clog_dir="/root/observer/store/clog", shared_mode=true, data_disk_total_size=99765694464, data_disk_default_percentage=60, clog_disk_total_size=99765694464, clog_disk_default_percentage=30) [2024-09-26 16:00:53.962491] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:361) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=6] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=99765694464, disk_size=15032385536) [2024-09-26 16:00:53.962495] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=4] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=99765694464) [2024-09-26 16:00:53.962503] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=7] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=99765694464, this={dir::"/root/observer/store/clog/log_pool", dir_fd:15, meta_fd:16, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:9388, max_block_id:9581, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2024-09-26 16:00:53.962510] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:133) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD7-0-0] [lt=7] refresh tenant config(tenants=[], ret=-5019) [2024-09-26 16:00:53.974000] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:53.974007] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:53.979845] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:53.984748] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:53.994357] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337653834439]}) [2024-09-26 16:00:54.004306] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:54.004314] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.028946] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500805, support_ssl=false) [2024-09-26 16:00:54.029013] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221500805, sess.client_addr_="127.0.0.1:36368") [2024-09-26 16:00:54.029020] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=5] accept one succ(*s={this:0x7f9267004030, session_id:3221500805, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:54.029107] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185924-0-0] [lt=49][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:54.029116] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185924-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:54.029124] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185924-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:54.029129] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185924-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:54.029137] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:54.029151] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=5] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:54.029296] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA3-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:54.029642] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA3-0-0] [lt=5][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:54.029650] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA3-0-0] [lt=6][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"(Zo@0nZW'Yl4ZMP,("r<"}, ret=-4043) [2024-09-26 16:00:54.029658] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA3-0-0] [lt=6][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:54.029663] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA3-0-0] [lt=4][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:54.029672] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA3-0-0] [lt=3] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:54.029697] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA3-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500805, proxy_sessid:0}) [2024-09-26 16:00:54.029703] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA3-0-0] [lt=5] mark session id unused(sessid=3221500805) [2024-09-26 16:00:54.029707] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA3-0-0] [lt=2][errcode=0] server close connection(sessid=3221500805, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:54.029711] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA3-0-0] [lt=4][errcode=-4018] get session fail(ret=-4018, sessid=3221500805, proxy_sessid=0) [2024-09-26 16:00:54.029714] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA3-0-0] [lt=3][errcode=-4016] session is null [2024-09-26 16:00:54.029718] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA3-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500805, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:54.029751] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f9267004030, session_id:3221500805, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337654028920, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:54.029760] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] kill and revert session(conn.sessid_=3221500805, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:54.029764] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9267004030, session_id:3221500805, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337654028920, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:54.029768] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500805, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:54.034529] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:54.034538] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.055595] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588F-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:54.055605] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588F-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:54.055609] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588F-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:54.055612] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588F-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:54.055615] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588F-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:54.055618] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588F-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:54.055621] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588F-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:54.055623] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588F-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:54.055625] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588F-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:54.055627] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588F-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:54.055628] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588F-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:54.055646] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8588F-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:54.058495] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:54.058502] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:54.058507] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337654058493) [2024-09-26 16:00:54.058512] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337653958389, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:54.058517] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:54.058525] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:54.058527] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:54.058772] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576A-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:54.058777] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576A-0-0] [lt=4][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:54.058794] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576A-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:54.058815] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337654058815, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=294, wlock_time=8, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:54.058822] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:54.058836] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223383294506, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:54.058840] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:54.058846] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:54.058850] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:54.058852] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:54.058857] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.058859] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.058861] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337654058825) [2024-09-26 16:00:54.065801] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:54.065806] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.065809] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:54.066074] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D76-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:54.066085] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D76-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:54.066104] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D76-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:54.087303] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=7] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:54.094796] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337653834439]}) [2024-09-26 16:00:54.096065] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:54.096072] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.096082] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:54.103020] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:54.103149] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=49, refresh_score_time=13, wash_time=2) [2024-09-26 16:00:54.103773] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=62914, clean_num=31457) [2024-09-26 16:00:54.115633] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 513.00) [2024-09-26 16:00:54.115640] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:54.115652] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:54.115655] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:54.126373] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:54.126379] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.132250] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337653132145]) [2024-09-26 16:00:54.132258] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337653132145], now0=[mts=1727337653132145], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:54.132264] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337653132068, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:54.132280] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:54.132287] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337653132068, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:54.132294] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337653132068, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:54.132310] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:54.132315] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:54.132317] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:54.132320] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:54.132322] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:54.132325] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=16, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:54.132329] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=16000, remain_us=12862003, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=16, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:54.149790] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:16, local_retry_times:16, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:54.149801] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:54.149804] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:54.149806] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:54.149839] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=21][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:54.149876] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:54.149879] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:54.154768] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-09-26 16:00:54.155085] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD6-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-09-26 16:00:54.155108] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD6-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:54.155133] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD6-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:54.156627] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:54.156632] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.158759] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:54.158768] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337654158757) [2024-09-26 16:00:54.158772] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337654058514, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:54.158787] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223383395068, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:54.158793] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:54.158800] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:54.158804] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:54.158807] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:54.158811] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.158813] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.158815] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337654158775) [2024-09-26 16:00:54.180086] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:54.186907] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:54.186917] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.194833] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337653834439]}) [2024-09-26 16:00:54.194849] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D942-0-0] [lt=15][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337654194826}) [2024-09-26 16:00:54.200575] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:00:54.200583] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=6] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:54.200592] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:54.217167] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:54.217175] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.247460] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:54.247469] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.250814] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:54.250827] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:54.250832] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:54.250847] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:54.250851] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:54.250853] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:54.250857] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:54.250860] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:54.250863] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:54.250865] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:54.250867] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:54.250871] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=3][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:54.250891] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589E-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:54.258908] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:54.258925] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337654258905) [2024-09-26 16:00:54.258930] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337654158774, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:54.258946] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223383494941, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:54.258952] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:54.258959] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:54.258964] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:54.258967] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:54.258972] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.258974] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.258975] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337654258933) [2024-09-26 16:00:54.277741] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:54.277748] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.295029] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337653834439]}) [2024-09-26 16:00:54.303935] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:54.304035] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=48, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:54.304658] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=5, clean_time=0, clean_start_pos=94371, clean_num=31457) [2024-09-26 16:00:54.307542] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:161, tid:2919}]) [2024-09-26 16:00:54.308009] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:54.308014] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.311917] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=22][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337654241919, now=[mts=1727337653311932]) [2024-09-26 16:00:54.311929] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337654241919, now=[mts=1727337653311932], now0=[mts=1727337653311932], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:54.311936] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269163240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337653311771, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:54.311955] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269163240}) [2024-09-26 16:00:54.311962] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269163240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337653311771, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:54.311970] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269163240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337653311771, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337652312160, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269163240}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:54.311986] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:54.311990] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:54.311993] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:54.311996] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:54.311998] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:54.312002] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:54.312007] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4] already timeout, do not need sleep(sleep_us=-87, remain_us=-87, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, timeout_timestamp=1727337654311919) [2024-09-26 16:00:54.312011] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:54.312018] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:54.312020] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:54.312021] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:54.312026] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:54.312070] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1764) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4012] get gts timeout(ret=-4012, expire_ts=1727337654241919, current_time=1727337654311914) [2024-09-26 16:00:54.312076] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4012] acquire global snapshot fail(ret=-4012, gts_ahead=0, expire_ts=1727337654241919, now=[mts=1727337654312070], now0=[mts=1727337654312070], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:54.312080] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4012] acquire global snapshot fail(ret=-4012, tx={this:0x7f9269163240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337654311914, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:54.312086] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=6][errcode=-4012] fail to get snapshot(ret=-4012, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269163240}) [2024-09-26 16:00:54.312090] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4012] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4012, session_id=1, *tx_desc={this:0x7f9269163240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337654311914, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:54.312095] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=5] start stmt(ret=-4012, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269163240, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337654311914, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337652312160, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269163240}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:54.312106] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=11][errcode=-4012] fail to start stmt(ret=-4012, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:54.312108] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4012] fail start stmt(ret=-4012) [2024-09-26 16:00:54.312109] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=1][errcode=-4012] execute plan failed(ret=-4012) [2024-09-26 16:00:54.312111] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4012] open result set failed(ret=-4012) [2024-09-26 16:00:54.312113] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=1][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:54.312115] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:54.312120] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:54.312123] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:54.312125] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:54.312126] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-09-26 16:00:54.312128] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C81-0-0] [lt=1][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:54.312131] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1999971) [2024-09-26 16:00:54.312134] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:54.312138] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:54.312141] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:54.312145] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:54.312148] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:54.312153] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:54.312174] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:54.312177] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:54.312182] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=4][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:54.312185] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:54.312188] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:54.312190] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:54.312297] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C81-0-0] [lt=3][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:54.312305] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:54.312307] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:54.312322] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:54.312326] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:54.312590] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:54.312598] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=8][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:54.335515] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.335527] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337654335512], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:54.335533] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337654335512], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:54.338335] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:54.338342] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.359789] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:54.359800] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337654359786) [2024-09-26 16:00:54.359805] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337654258933, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:54.359823] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223383595321, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:54.359828] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:54.359835] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:54.359840] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:54.359842] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:54.359847] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.359849] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.359851] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337654359808) [2024-09-26 16:00:54.369547] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:54.369555] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.380296] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:54.390095] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=6] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:54.390103] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [2645][Blacklist][T0][Y0-0000000000000000-0-0] [lt=11] blacklist_loop exec finished(cost_time=11, is_enabled=true, send_cnt=0) [2024-09-26 16:00:54.390108] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=2] current monitor number(seq_=-1) [2024-09-26 16:00:54.395943] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337654335512]}) [2024-09-26 16:00:54.399819] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:54.399827] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.403810] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D3FDA-0-0] [lt=12][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:54.429144] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337654429143, bytes: 14683534, bw: 0.003349 MB/s, add_ts: 1008164, add_bytes: 3540 [2024-09-26 16:00:54.431087] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:54.431100] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.460401] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:54.460424] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337654460388) [2024-09-26 16:00:54.460430] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337654359808, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:54.460449] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223383696031, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:54.460476] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:54.460485] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:54.460491] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:54.460494] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:54.460501] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.460503] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.460505] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337654460433) [2024-09-26 16:00:54.461194] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:54.461204] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.462059] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.462065] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.462069] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.462080] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.462082] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.462084] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.462679] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.462683] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.462685] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.462695] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.462697] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.462699] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.463660] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.463665] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.463669] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.463676] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.463678] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.463680] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.464482] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.464489] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.464493] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.464499] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.464501] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.464503] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.465793] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.465797] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.465800] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.465810] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.465813] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.465814] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.466610] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.466613] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.466615] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.466623] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.466625] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.466627] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.467224] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.467233] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.467238] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.467242] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.467244] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.467246] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.467989] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.467996] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.468000] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.468004] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.468006] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.468008] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:54.468648] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.468661] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.469215] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.469221] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.469926] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.469935] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.470564] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.470573] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.471086] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.471187] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.471686] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.471921] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.472306] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.472536] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.472917] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.473101] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.473463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.473688] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.474081] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.474265] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.474658] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.474781] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.476025] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.476044] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.477021] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.477030] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.478021] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.478029] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.478996] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.479003] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.479933] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.479947] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.480899] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.480909] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.481865] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.481875] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.482875] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.482884] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.483830] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.483842] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.484823] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.484824] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.485773] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.486772] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.486806] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.487741] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.488746] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.488756] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.490695] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.490695] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.491645] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.491681] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.491707] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:54.491718] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.493589] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.493626] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.494557] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.494606] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.495531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.495568] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.496529] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.496533] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.496540] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337654335512]}) [2024-09-26 16:00:54.497511] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.497511] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.498507] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.498515] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.499468] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.499471] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.500443] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.500458] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.501453] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.501463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.502419] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.502400] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.503381] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.503381] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.504322] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.504346] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.505326] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.505347] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.505450] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:54.505574] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=52, refresh_score_time=36, wash_time=2) [2024-09-26 16:00:54.506178] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=5, clean_time=0, clean_start_pos=125828, clean_num=31457) [2024-09-26 16:00:54.506258] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.506270] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.507282] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.507296] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.508265] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.508272] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:54.521927] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:00:54.521934] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.523018] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185925-0-0] [lt=59][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:54.523036] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185925-0-0] [lt=17][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:54.523045] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185925-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:54.523051] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185925-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:54.546397] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=1, size_used=6558, mem_used=12478464) [2024-09-26 16:00:54.552203] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=35] success to notify_need_writing_throttling True [2024-09-26 16:00:54.552213] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.561014] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:54.561037] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337654561010) [2024-09-26 16:00:54.561043] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337654460433, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:54.561049] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:54.561060] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:54.561063] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:54.561388] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:54.561400] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:54.561404] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:54.561409] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:54.561413] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:54.561416] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:54.561431] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=13][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:54.561434] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:54.561436] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:54.561438] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:54.561440] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:54.561442] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:54.561445] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:54.561447] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:54.561456] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=6][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:54.561459] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:54.561462] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:54.561464] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:54.561467] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:54.561469] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:54.561472] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:54.561477] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:54.561483] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:54.561486] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:54.561487] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:54.561493] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:54.561515] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=21][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:54.561519] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576B-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:54.561522] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:54.561525] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:54.561528] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:54.561532] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337654561313, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:54.561536] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:54.561539] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:54.561567] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:54.561571] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337654561570, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=514, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:54.561578] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:54.561581] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:54.561598] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223383797766, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:54.561603] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:54.561609] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:54.561614] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:54.561616] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:54.561622] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.561624] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.561626] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337654561585) [2024-09-26 16:00:54.564910] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=6] gc stale ls task succ [2024-09-26 16:00:54.580525] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:54.583473] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:54.583483] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.597087] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337654335512]}) [2024-09-26 16:00:54.601074] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855EF-0-0] [lt=18] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:54.602038] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.602039] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.602043] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.602060] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.614713] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:54.614720] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.644999] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:54.645008] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.662583] WDIAG [PALF] convert_to_ts (scn.cpp:265) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2024-09-26 16:00:54.662593] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:54.662590] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1727337654662581, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1727337654662581, force_self_check=false, weak_read_refresh_interval=100000) [2024-09-26 16:00:54.662595] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:54.662606] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:54.662612] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337654662604) [2024-09-26 16:00:54.662616] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337654561046, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:54.662635] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223383899063, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:54.662641] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:54.662649] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:54.662654] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:54.662656] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:54.662661] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.662663] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.662665] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337654662619) [2024-09-26 16:00:54.675306] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:54.675322] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.697537] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337654335512]}) [2024-09-26 16:00:54.705442] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:54.705452] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.706372] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:54.706502] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash time detail, (compute_wash_size_time=58, refresh_score_time=16, wash_time=1) [2024-09-26 16:00:54.707094] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=5, clean_time=0, clean_start_pos=157285, clean_num=31457) [2024-09-26 16:00:54.735822] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:54.735835] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.736813] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=39] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:54.740730] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:54.760231] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=12] XA scheduler heartbeat task statistics(avg_time=7) [2024-09-26 16:00:54.763174] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:54.763206] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337654763171) [2024-09-26 16:00:54.763212] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337654662619, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:54.763226] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get_max_applied_scn(scn={val:1727114270763524000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:374, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:1727114270763524000, v:0}, max_applied_cb_scn_:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:54.763235] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] get_min_unreplayed_log_info(lsn={lsn:629872191939}, scn={val:1727114270763524001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:00:54.763255] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=223383999640, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:54.763261] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:54.763268] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:54.763273] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:54.763276] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:54.763293] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.763296] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.763301] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337654763215) [2024-09-26 16:00:54.766115] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14] success to notify_need_writing_throttling True [2024-09-26 16:00:54.766124] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.780754] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:54.782727] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=4] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-09-26 16:00:54.782736] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-09-26 16:00:54.782750] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=5] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=691) [2024-09-26 16:00:54.796416] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:54.796426] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.798360] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337654335512]}) [2024-09-26 16:00:54.821781] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.821777] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.821807] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.821813] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.821817] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.821815] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.821821] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.821828] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.822745] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.822772] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.822783] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:54.826683] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:00:54.826699] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.836564] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:54.836574] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337654836562], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:54.836581] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337654836562], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:54.836585] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:54.836589] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:54.856941] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:54.856948] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.863769] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:54.863779] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337654863765) [2024-09-26 16:00:54.863784] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337654763215, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:54.863800] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223384099293, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:54.863807] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:54.863814] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:54.863820] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:54.863823] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:54.863829] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.863831] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.863833] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337654863787) [2024-09-26 16:00:54.867682] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337654867681, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008205, add_bytes: 0 [2024-09-26 16:00:54.872919] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, table_name.ptr()="data_size:16, data:5F5F616C6C5F6D657267655F696E666F", ret=-5019) [2024-09-26 16:00:54.872930] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, ret=-5019) [2024-09-26 16:00:54.872934] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_merge_info, db_name=oceanbase) [2024-09-26 16:00:54.872937] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_merge_info) [2024-09-26 16:00:54.872941] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:54.872944] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:54.872947] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_merge_info' doesn't exist [2024-09-26 16:00:54.872949] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:54.872951] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:54.872953] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:54.872955] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:54.872957] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:54.872959] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:54.872962] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:54.872967] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:54.872970] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:54.872973] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:54.872975] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:54.872977] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_merge_info WHERE tenant_id = '1', ret=-5019) [2024-09-26 16:00:54.872979] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:54.872982] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:54.872987] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:54.872992] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:54.872994] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:54.872995] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:54.873001] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:54.873004] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:54.873006] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885088-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, aret=-5019, ret=-5019) [2024-09-26 16:00:54.873008] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:54.873011] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:54.873013] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:54.873016] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337654872853, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:54.873018] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:54.873020] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:49) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:54.873049] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:890) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] failed to load global merge info(ret=-5019, ret="OB_TABLE_NOT_EXIST", global_merge_info={tenant_id:1, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2024-09-26 16:00:54.873060] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1005) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] fail to refresh merge info(tmp_ret=-5019, tmp_ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:54.873264] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:54.873267] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:54.887252] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:54.887261] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.898927] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337654836562]}) [2024-09-26 16:00:54.907866] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:54.907977] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=53, refresh_score_time=23, wash_time=1) [2024-09-26 16:00:54.908567] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=188742, clean_num=31457) [2024-09-26 16:00:54.917490] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:54.917498] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.919511] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] ====== checkpoint timer task ====== [2024-09-26 16:00:54.919915] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] succ to get rec scn(*this={this:0x7f92a6a04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1726077600526835000, v:0}, last_push_gc_task_ts_:1727334198745636, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:54.919929] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13] tx ctx memtable get rec scn(this={ObITable:{this:0x7f92a22a4080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f92a22a4080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:54.919941] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1723174350926847000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-09-26 16:00:54.919969] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:54.919994] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}, min_rec_scn={val:1726163586617842000, v:0}, ls_id_={id:1}) [2024-09-26 16:00:54.920008] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:54.920011] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:54.920013] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=1] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:54.920032] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1726163586617842000, v:0}, checkpoint_scn_in_ls_meta={val:1726163586617842000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-09-26 16:00:54.920043] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}, curr_base_lsn={lsn:627966418944}, new_base_lsn={lsn:627966418944}, lsn={lsn:627966418944}) [2024-09-26 16:00:54.920048] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:627966418944}, ls->get_ls_id()={id:1}) [2024-09-26 16:00:54.920052] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-09-26 16:00:54.933416] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2024-09-26 16:00:54.933428] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2024-09-26 16:00:54.933432] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2024-09-26 16:00:54.933435] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2024-09-26 16:00:54.933439] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:54.933441] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:54.933444] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2024-09-26 16:00:54.933446] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:54.933448] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:54.933450] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:54.933452] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:54.933454] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:54.933456] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:54.933458] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:54.933464] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:54.933466] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:54.933468] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:54.933470] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:54.933473] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", ret=-5019) [2024-09-26 16:00:54.933481] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=7][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:54.933483] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:54.933503] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:54.933511] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=7][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:54.933513] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:54.933515] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:54.933520] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:54.933524] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:54.933527] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C43-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%""}, aret=-5019, ret=-5019) [2024-09-26 16:00:54.933529] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-09-26 16:00:54.933532] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:54.933534] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:54.933537] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337654933359, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-09-26 16:00:54.933541] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:54.933543] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:945) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] failed to read(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%") [2024-09-26 16:00:54.933569] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:674) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] fail to inner get ls status info(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * FROM __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%", tenant_id=1, exec_tenant_id=1, need_member_list=false) [2024-09-26 16:00:54.933573] WDIAG [STORAGE.DUP_TABLE] refresh_dup_tablet_schema_ (ob_dup_table_util.cpp:97) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] get duplicate ls status info failed(ret=-5019, dup_ls_status_info={tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2024-09-26 16:00:54.933579] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:148) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] refresh dup table schema failed(ret=-5019) [2024-09-26 16:00:54.933581] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:236) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] scan dup ls to find dup_tablet failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:54.933584] WDIAG [STORAGE.DUP_TABLE] runTimerTask (ob_dup_table_util.cpp:73) [2904][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute dup ls scan failed(tmp_ret=-5019) [2024-09-26 16:00:54.947770] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:54.947779] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.964405] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:54.964414] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337654964403) [2024-09-26 16:00:54.964419] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337654863787, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:54.964445] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223384199921, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:54.964453] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:54.964473] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:54.964478] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:54.964481] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:54.964487] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.964490] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:54.964491] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337654964422) [2024-09-26 16:00:54.978055] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:54.978063] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:54.980983] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:54.984937] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:54.985927] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-09-26 16:00:54.985936] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=7] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:54.985940] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=4] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:54.985943] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=3] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:54.987641] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] Dumping All Cache Objs(alloc_obj_list.count()=3, alloc_obj_list=[{obj_id:78, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157779}, {obj_id:79, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:5, added_to_lc:true, mem_used:175172}]) [2024-09-26 16:00:54.987653] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=11] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:54.999546] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337654836562]}) [2024-09-26 16:00:55.008332] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.008339] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.027094] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185926-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:55.027103] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185926-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:55.027111] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185926-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:55.027116] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185926-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:55.030022] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221500813, support_ssl=false) [2024-09-26 16:00:55.030064] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221500813, sess.client_addr_="127.0.0.1:36370") [2024-09-26 16:00:55.030071] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f9264004030, session_id:3221500813, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:55.030091] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:55.030099] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:55.030102] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:280) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=3] mysql login queue(mysql_queue->queue_.size()=1) [2024-09-26 16:00:55.030136] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA4-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:55.030337] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA4-0-0] [lt=6][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:55.030346] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA4-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"HO5[JBK1S18dn}I4e#|h"}, ret=-4043) [2024-09-26 16:00:55.030353] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA4-0-0] [lt=6][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:55.030357] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA4-0-0] [lt=4][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:55.030367] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA4-0-0] [lt=3] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:55.030390] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA4-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500813, proxy_sessid:0}) [2024-09-26 16:00:55.030394] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA4-0-0] [lt=4] mark session id unused(sessid=3221500813) [2024-09-26 16:00:55.030399] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA4-0-0] [lt=4][errcode=0] server close connection(sessid=3221500813, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:55.030403] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA4-0-0] [lt=4][errcode=-4018] get session fail(ret=-4018, sessid=3221500813, proxy_sessid=0) [2024-09-26 16:00:55.030406] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA4-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:55.030411] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA4-0-0] [lt=4] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500813, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:55.030447] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4015] close sql sock by user req(*s={this:0x7f9264004030, session_id:3221500813, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337655029830, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:55.030477] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=28] kill and revert session(conn.sessid_=3221500813, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:55.030481] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] can close safely, do destroy(*s={this:0x7f9264004030, session_id:3221500813, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337655029830, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:55.030486] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500813, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:55.038599] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:55.038607] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.056505] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:55.056518] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:55.056522] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:55.056526] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:55.056529] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:55.056531] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:55.056535] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:55.056537] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:55.056539] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:55.056541] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:55.056542] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:55.056544] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:55.056547] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:55.056549] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:55.056573] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=20][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:55.056576] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:55.056579] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:55.056585] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:55.056588] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:55.056590] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:55.056593] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:55.056599] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:55.056604] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:55.056607] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:55.056608] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:55.056614] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:55.056617] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85890-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:55.056621] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85890-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:55.056624] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85890-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:55.056627] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:55.056631] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85890-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:55.056634] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85890-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337655056436, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:55.056637] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85890-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:55.056639] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85890-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:55.056662] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85890-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:55.056666] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85890-0-0] [lt=3][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:55.065961] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:55.065969] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:55.065975] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337655065959) [2024-09-26 16:00:55.065980] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337654964422, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:55.065994] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:55.066003] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:55.066006] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:55.066170] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=6] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:55.066292] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:55.066300] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:55.066305] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:55.066308] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:55.066311] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:55.066313] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:55.066316] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:55.066318] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:55.066319] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:55.066322] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:55.066323] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:55.066325] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:55.066327] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:55.066329] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:55.066335] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:55.066337] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:55.066340] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:55.066342] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:55.066344] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:55.066346] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:55.066348] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:55.066352] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:55.066356] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:55.066359] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:55.066360] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:55.066365] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:55.066369] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:55.066371] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576C-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:55.066373] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:55.066376] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:55.066378] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:55.066380] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337655066237, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:55.066383] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:55.066385] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:55.066407] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:55.066410] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337655066409, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=410, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:55.066416] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:55.066418] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:55.066435] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223384302445, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.066435] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:55.066442] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:55.066440] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:55.066445] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:55.066446] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:55.066449] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:55.066451] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:55.066452] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:55.066453] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:55.066454] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:55.066456] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:55.066458] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:55.066458] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.066460] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:55.066461] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:55.066461] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.066463] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:55.066463] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337655066422) [2024-09-26 16:00:55.066465] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:55.066466] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:55.066468] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:55.066472] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:55.066474] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:55.066475] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:55.066478] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:55.066480] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:55.066482] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:55.066484] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:55.066487] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:55.066491] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:55.066493] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:55.066494] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:55.066497] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:55.066500] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:55.066502] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D78-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:55.066504] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:55.066506] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.069900] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.069905] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.087449] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:55.100114] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.100124] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.100135] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:55.100204] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=33] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337654836562]}) [2024-09-26 16:00:55.109072] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:55.109185] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=52, refresh_score_time=24, wash_time=1) [2024-09-26 16:00:55.109801] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=220199, clean_num=31457) [2024-09-26 16:00:55.120679] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10638.00) [2024-09-26 16:00:55.120689] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:55.120712] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:55.120715] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:55.130463] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:55.130470] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.149971] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=24][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337654149875]) [2024-09-26 16:00:55.149982] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337654149875], now0=[mts=1727337654149875], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:55.149989] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337654148817, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:55.150005] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:55.150013] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337654148817, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:55.150020] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337654148817, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:55.150036] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:55.150041] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:55.150055] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:55.150057] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:55.150059] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:55.150063] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=17, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:55.150067] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] will sleep(sleep_us=17000, remain_us=11844266, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:55.160741] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.160748] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.166558] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:55.166569] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337655166556) [2024-09-26 16:00:55.166574] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337655065983, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:55.166588] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223384402170, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.166594] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:55.166601] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:55.166606] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:55.166608] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:55.166613] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.166615] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.166617] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337655166577) [2024-09-26 16:00:55.167522] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:17, local_retry_times:17, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:55.167532] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:55.167536] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:55.167537] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:55.167543] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:55.167581] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:55.167586] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:55.181116] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:55.190917] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:55.190927] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.200721] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337654836562]}) [2024-09-26 16:00:55.200738] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D94E-0-0] [lt=18][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337655200713}) [2024-09-26 16:00:55.208777] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:55.208790] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:55.208796] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:55.221231] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:55.221240] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.251389] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589F-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:55.251404] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589F-0-0] [lt=13][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.251433] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-000623005648589F-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:55.251482] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:55.251491] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.267160] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:55.267177] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337655267157) [2024-09-26 16:00:55.267183] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337655166576, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:55.267200] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223384502694, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.267206] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:55.267213] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:55.267218] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:55.267220] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:55.267225] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.267227] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.267229] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337655267186) [2024-09-26 16:00:55.281833] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:55.281840] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.301348] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337654836562]}) [2024-09-26 16:00:55.310246] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:55.310361] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=52, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:55.310984] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=251656, clean_num=31457) [2024-09-26 16:00:55.312082] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337656242330, now=[mts=1727337654312589]) [2024-09-26 16:00:55.312087] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:55.312093] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337656242330, now=[mts=1727337654312589], now0=[mts=1727337654312589], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:55.312100] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.312100] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269164fe0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337654311914, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:55.312116] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269164fe0}) [2024-09-26 16:00:55.312124] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269164fe0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337654311914, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:55.312132] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269164fe0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337654311914, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337654312572, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269164fe0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:55.312148] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:55.312152] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:55.312156] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:55.312160] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:55.312163] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:55.312166] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:55.312183] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=16] already timeout, do not need sleep(sleep_us=0, remain_us=1000147, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337656312330) [2024-09-26 16:00:55.312187] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:55.312195] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:55.312197] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:55.312198] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:55.312203] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:55.312236] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:00:55.312246] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=10][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:55.335391] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=28] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:55.336912] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.336924] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337655336910], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:55.336931] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337655336910], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:55.342387] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:55.342394] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.367787] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:55.367798] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337655367784) [2024-09-26 16:00:55.367802] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337655267186, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:55.367819] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223384604281, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.367824] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:55.367831] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:55.367836] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:55.367839] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:55.367846] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.367848] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.367850] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337655367805) [2024-09-26 16:00:55.372636] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.372642] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.381750] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=833584, replace_num=15728) [2024-09-26 16:00:55.381760] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:55.390209] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:55.401927] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337655336910]}) [2024-09-26 16:00:55.403912] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.403919] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.407841] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D4022-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:55.411985] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F52-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:55.411999] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F52-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:55.412006] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F52-0-0] [lt=5][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.412011] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F52-0-0] [lt=0] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:55.412036] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F52-0-0] [lt=0] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:55.434055] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.434063] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.436124] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=40] PNIO [ratelimit] time: 1727337655436123, bytes: 14687356, bw: 0.003620 MB/s, add_ts: 1006980, add_bytes: 3822 [2024-09-26 16:00:55.462772] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.462781] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.462786] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.462792] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.462795] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.462797] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.463448] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.463470] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.463474] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=25][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.463478] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.463480] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.463483] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.463489] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.463491] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.463493] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.464461] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.464467] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.464480] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.464489] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.464492] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.464494] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.464497] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.464498] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.464500] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.464500] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.464502] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.465454] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.465458] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.465460] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.465460] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.465463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.465465] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.465471] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.465473] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.465475] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.466389] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.466395] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.466397] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.466452] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.466455] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.466456] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.466470] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.466473] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.466474] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.467382] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.467387] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.467389] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.467393] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.467394] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.467405] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:55.467421] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] ====== tenant freeze timer task ====== [2024-09-26 16:00:55.467422] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.467428] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:55.467472] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337650629877, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337650629842, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337650629895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:55.467497] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:55.467549] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E0-0-0] [lt=3] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:55.467555] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E0-0-0] [lt=4] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:55.467559] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E0-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:55.467563] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E0-0-0] [lt=3] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:55.467566] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E0-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:55.467571] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E0-0-0] [lt=4][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:55.467573] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E0-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:55.467593] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E0-0-0] [lt=20] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:55.467596] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E0-0-0] [lt=3][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:55.467598] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E0-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:55.467601] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E0-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:55.467631] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851E0-0-0] [lt=5] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:55.468339] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.468355] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.468360] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.468369] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:55.468377] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337655468366) [2024-09-26 16:00:55.468382] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337655367805, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:55.468399] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223384704113, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.468404] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:55.468411] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:55.468416] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:55.468418] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:55.468423] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.468425] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.468427] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337655468385) [2024-09-26 16:00:55.469309] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.469335] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.469340] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.470303] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.470360] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.470368] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.471258] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.471271] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.471278] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.471816] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.471826] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.471831] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.473226] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.473233] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.473238] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.474205] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.474223] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.474239] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.475226] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.475232] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.475596] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.475815] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.475954] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.476152] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.476362] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.476483] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.477188] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.477322] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.477329] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.477806] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.477834] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.478060] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.478365] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.478375] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.478707] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.478877] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.478998] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.479323] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.479460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.479577] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.479882] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.480050] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.480266] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.480521] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.480623] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.480833] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.481064] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.481194] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.481386] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.481608] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.481823] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.481968] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.482190] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.482411] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.482529] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.482857] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.482959] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.483976] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.483982] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.483986] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.484949] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.484976] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.484992] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.485496] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.485622] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.485627] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.486514] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.486519] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.486531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.487116] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.487295] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.487308] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.487662] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.487834] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.487844] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:55.488506] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=9][errcode=-4038] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.488682] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=8][errcode=-4038] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.488695] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.494621] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:55.494631] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.502520] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337655336910]}) [2024-09-26 16:00:55.511430] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:55.511555] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash time detail, (compute_wash_size_time=55, refresh_score_time=15, wash_time=1) [2024-09-26 16:00:55.512165] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=283113, clean_num=31457) [2024-09-26 16:00:55.513316] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4038, dropped:5, tid:2547}]) [2024-09-26 16:00:55.521207] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185927-0-0] [lt=31][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:55.521220] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185927-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:55.521229] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185927-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:55.521234] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185927-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:55.525043] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:55.525052] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.546468] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6568, mem_used=12478464) [2024-09-26 16:00:55.555274] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:55.555281] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.565039] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:00:55.568960] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:55.568971] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337655568957) [2024-09-26 16:00:55.568976] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337655468385, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:55.568981] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:55.568992] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:55.568995] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:55.569331] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576D-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:55.569339] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576D-0-0] [lt=7][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.569368] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576D-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:55.569401] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337655569399, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=413, wlock_time=11, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:55.569410] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:55.569426] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223384805505, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.569431] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:55.569438] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:55.569443] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:55.569445] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:55.569451] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.569453] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.569455] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337655569414) [2024-09-26 16:00:55.582174] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=4] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:55.585397] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:55.585413] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.590476] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [2619][DumpLSLoc][T0][YB42C0A8002D-0006230058384C7B-0-0] [lt=20] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1727337650934512, replica_locations:[{server:"192.168.0.45:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:374}]}]) [2024-09-26 16:00:55.602180] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.602183] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [2635][LocalityReload][T0][Y0-0000000000000000-0-0] [lt=10] runTimer to refresh locality_info(ret=0) [2024-09-26 16:00:55.602194] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.602199] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=12] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.602206] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.602527] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7B-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:55.602536] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7B-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.602560] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7B-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:55.602566] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=0] load region(ret=-5019, locality_info={version:0, local_region:"", local_zone:"", local_idc:"", local_zone_type:3, local_zone_status:3, locality_region_array:[], locality_zone_array:[]}) [2024-09-26 16:00:55.603032] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337655336910]}) [2024-09-26 16:00:55.613449] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:142, tid:2932}]) [2024-09-26 16:00:55.615847] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.615853] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.631493] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] ====== traversal_flush timer task ====== [2024-09-26 16:00:55.631525] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f927282a030, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6163825430799493735, dag_cnt=1, dag_type_cnts=1) [2024-09-26 16:00:55.631533] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] success to create and add dag(ret=0, dag=0x7f927282a030) [2024-09-26 16:00:55.631536] INFO [STORAGE] flush (ob_tx_data_memtable.cpp:917) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] [TX DATA MERGE]schedule flush tx data memtable task done(ret=0, ret="OB_SUCCESS", get_ls_id()={id:1}, this=0x7f92a2294080, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:4, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337655631535, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:55.631572] INFO [MDS] flush (mds_table_mgr.cpp:172) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=20] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:55.631607] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=21] add dag success(dag=0x7f927282a560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6659188001345271846, dag_cnt=2, dag_type_cnts=1) [2024-09-26 16:00:55.631609] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927282a560) [2024-09-26 16:00:55.631613] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, create_flush_dag_time:1727337655631612, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452867, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200240}, checkpoint_unit_ptr:0x7f92a1a70850}, is_freeze_clock:true}) [2024-09-26 16:00:55.631620] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337655631601, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631635] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=12] add dag success(dag=0x7f927282aa90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-9069613569758001279, dag_cnt=3, dag_type_cnts=2) [2024-09-26 16:00:55.631637] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927282aa90) [2024-09-26 16:00:55.631639] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, create_flush_dag_time:1727337655631638, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452892, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200489}, checkpoint_unit_ptr:0x7f92a1a70080}, is_freeze_clock:true}) [2024-09-26 16:00:55.631642] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337655631632, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631651] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927282afc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1288986994123435108, dag_cnt=4, dag_type_cnts=3) [2024-09-26 16:00:55.631653] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927282afc0) [2024-09-26 16:00:55.631655] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337655631654, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452911, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847222}, checkpoint_unit_ptr:0x7f92a1a71020}, is_freeze_clock:true}) [2024-09-26 16:00:55.631658] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337655631649, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631666] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927282b4f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5355160120942094702, dag_cnt=5, dag_type_cnts=4) [2024-09-26 16:00:55.631668] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927282b4f0) [2024-09-26 16:00:55.631669] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337655631669, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452929, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847223}, checkpoint_unit_ptr:0x7f92a1a717f0}, is_freeze_clock:true}) [2024-09-26 16:00:55.631673] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337655631664, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631682] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405042) [2024-09-26 16:00:55.631693] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:55.631697] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405070) [2024-09-26 16:00:55.631704] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:55.631706] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405087) [2024-09-26 16:00:55.631713] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:55.631716] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405104) [2024-09-26 16:00:55.631722] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:00:55.631726] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] add dag success(dag=0x7f927282ba20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1994541446894784994, dag_cnt=6, dag_type_cnts=5) [2024-09-26 16:00:55.631727] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f927282ba20) [2024-09-26 16:00:55.631729] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, create_flush_dag_time:1727337655631729, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209556884, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:226}, checkpoint_unit_ptr:0x7f92a1a73f00}, is_freeze_clock:true}) [2024-09-26 16:00:55.631733] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337655631724, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631745] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927282bf50, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8791616776033068452, dag_cnt=7, dag_type_cnts=6) [2024-09-26 16:00:55.631747] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927282bf50) [2024-09-26 16:00:55.631748] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337655631748, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405141, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847408}, checkpoint_unit_ptr:0x7f92a1a746d0}, is_freeze_clock:true}) [2024-09-26 16:00:55.631752] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337655631739, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631763] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927282c480, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6550230011160095889, dag_cnt=8, dag_type_cnts=7) [2024-09-26 16:00:55.631765] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927282c480) [2024-09-26 16:00:55.631767] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337655631767, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405158, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847409}, checkpoint_unit_ptr:0x7f92a1a75670}, is_freeze_clock:true}) [2024-09-26 16:00:55.631770] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337655631758, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631778] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927282ca30, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6809498982591245614, dag_cnt=9, dag_type_cnts=8) [2024-09-26 16:00:55.631780] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927282ca30) [2024-09-26 16:00:55.631781] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, create_flush_dag_time:1727337655631781, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405174, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:200004}, checkpoint_unit_ptr:0x7f92a1a74ea0}, is_freeze_clock:true}) [2024-09-26 16:00:55.631784] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337655631776, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631793] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927282cf60, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6815944285528637599, dag_cnt=10, dag_type_cnts=9) [2024-09-26 16:00:55.631795] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927282cf60) [2024-09-26 16:00:55.631797] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, create_flush_dag_time:1727337655631796, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209506214, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1}, checkpoint_unit_ptr:0x7f92a1a76610}, is_freeze_clock:true}) [2024-09-26 16:00:55.631800] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337655631791, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631808] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927282d490, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-4182959058374342994, dag_cnt=11, dag_type_cnts=10) [2024-09-26 16:00:55.631809] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f927282d490) [2024-09-26 16:00:55.631812] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, create_flush_dag_time:1727337655631812, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405204, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:412}, checkpoint_unit_ptr:0x7f92a1a75e40}, is_freeze_clock:true}) [2024-09-26 16:00:55.631815] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337655631806, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631824] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927282d9c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1586518267935340, dag_cnt=12, dag_type_cnts=11) [2024-09-26 16:00:55.631825] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f927282d9c0) [2024-09-26 16:00:55.631827] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, create_flush_dag_time:1727337655631827, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405220, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:444}, checkpoint_unit_ptr:0x7f92a1a775b0}, is_freeze_clock:true}) [2024-09-26 16:00:55.631830] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337655631822, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631838] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927282def0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3044367941103776476, dag_cnt=13, dag_type_cnts=12) [2024-09-26 16:00:55.631840] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927282def0) [2024-09-26 16:00:55.631841] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, create_flush_dag_time:1727337655631841, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455964, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:309}, checkpoint_unit_ptr:0x7f92a1a76de0}, is_freeze_clock:true}) [2024-09-26 16:00:55.631844] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337655631836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631853] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927282e420, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1596830851104282047, dag_cnt=14, dag_type_cnts=13) [2024-09-26 16:00:55.631854] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f927282e420) [2024-09-26 16:00:55.631856] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, create_flush_dag_time:1727337655631856, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455986, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:331}, checkpoint_unit_ptr:0x7f92a1a77d80}, is_freeze_clock:true}) [2024-09-26 16:00:55.631859] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337655631851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631867] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927282e950, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2360195962081676673, dag_cnt=15, dag_type_cnts=14) [2024-09-26 16:00:55.631869] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927282e950) [2024-09-26 16:00:55.631870] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, create_flush_dag_time:1727337655631870, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405262, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:458}, checkpoint_unit_ptr:0x7f92a1a7a490}, is_freeze_clock:true}) [2024-09-26 16:00:55.631873] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337655631865, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631882] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927282ee80, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5675615614797020460, dag_cnt=16, dag_type_cnts=15) [2024-09-26 16:00:55.631883] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f927282ee80) [2024-09-26 16:00:55.631885] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, create_flush_dag_time:1727337655631885, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405279, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:487}, checkpoint_unit_ptr:0x7f92a1a79cc0}, is_freeze_clock:true}) [2024-09-26 16:00:55.631888] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337655631880, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631897] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927282f3b0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-612435342908088006, dag_cnt=17, dag_type_cnts=16) [2024-09-26 16:00:55.631898] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f927282f3b0) [2024-09-26 16:00:55.631900] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, create_flush_dag_time:1727337655631900, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405296, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:455}, checkpoint_unit_ptr:0x7f92a1a794f0}, is_freeze_clock:true}) [2024-09-26 16:00:55.631903] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337655631894, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631911] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f927282f8e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7868818205990873505, dag_cnt=18, dag_type_cnts=17) [2024-09-26 16:00:55.631913] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927282f8e0) [2024-09-26 16:00:55.631915] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, create_flush_dag_time:1727337655631914, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405313, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:492}, checkpoint_unit_ptr:0x7f92a1a78550}, is_freeze_clock:true}) [2024-09-26 16:00:55.631917] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337655631909, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631926] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f927282fe10, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7071731214611448811, dag_cnt=19, dag_type_cnts=18) [2024-09-26 16:00:55.631928] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f927282fe10) [2024-09-26 16:00:55.631929] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, create_flush_dag_time:1727337655631929, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405347, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:501}, checkpoint_unit_ptr:0x7f92a1a78d20}, is_freeze_clock:true}) [2024-09-26 16:00:55.631932] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337655631924, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631941] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272830340, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8731750006000903454, dag_cnt=20, dag_type_cnts=19) [2024-09-26 16:00:55.631943] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272830340) [2024-09-26 16:00:55.631944] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, create_flush_dag_time:1727337655631944, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405365, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:472}, checkpoint_unit_ptr:0x7f92a1a7ac60}, is_freeze_clock:true}) [2024-09-26 16:00:55.631947] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337655631939, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631956] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272830870, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6766388540859538791, dag_cnt=21, dag_type_cnts=20) [2024-09-26 16:00:55.631957] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272830870) [2024-09-26 16:00:55.631959] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, create_flush_dag_time:1727337655631959, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405384, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:342}, checkpoint_unit_ptr:0x7f92a1a7cba0}, is_freeze_clock:true}) [2024-09-26 16:00:55.631962] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337655631954, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631970] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272830da0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2986199386776543176, dag_cnt=22, dag_type_cnts=21) [2024-09-26 16:00:55.631971] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272830da0) [2024-09-26 16:00:55.631973] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337655631973, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405418, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:140}, checkpoint_unit_ptr:0x7f92a1a7b430}, is_freeze_clock:true}) [2024-09-26 16:00:55.631976] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337655631968, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631984] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f92728312d0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6947063304478012270, dag_cnt=23, dag_type_cnts=22) [2024-09-26 16:00:55.631986] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f92728312d0) [2024-09-26 16:00:55.631987] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337655631987, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405436, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101009}, checkpoint_unit_ptr:0x7f92a1a7bc00}, is_freeze_clock:true}) [2024-09-26 16:00:55.631990] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337655631982, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.631998] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272831800, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5843378789243662202, dag_cnt=24, dag_type_cnts=23) [2024-09-26 16:00:55.632000] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272831800) [2024-09-26 16:00:55.632001] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337655632001, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405453, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101010}, checkpoint_unit_ptr:0x7f92a1a7c3d0}, is_freeze_clock:true}) [2024-09-26 16:00:55.632004] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337655631996, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632013] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272831d30, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1841823684513746176, dag_cnt=25, dag_type_cnts=24) [2024-09-26 16:00:55.632015] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272831d30) [2024-09-26 16:00:55.632016] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, create_flush_dag_time:1727337655632016, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405470, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:323}, checkpoint_unit_ptr:0x7f92a1a7d370}, is_freeze_clock:true}) [2024-09-26 16:00:55.632019] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337655632011, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632027] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272832260, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-2678407261265885087, dag_cnt=26, dag_type_cnts=25) [2024-09-26 16:00:55.632029] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272832260) [2024-09-26 16:00:55.632030] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, create_flush_dag_time:1727337655632030, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405487, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:377}, checkpoint_unit_ptr:0x7f92a1a7e310}, is_freeze_clock:true}) [2024-09-26 16:00:55.632033] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337655632025, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632041] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272832790, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7175089551615529855, dag_cnt=27, dag_type_cnts=26) [2024-09-26 16:00:55.632043] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272832790) [2024-09-26 16:00:55.632045] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, create_flush_dag_time:1727337655632045, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405503, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:297}, checkpoint_unit_ptr:0x7f92a1a7eae0}, is_freeze_clock:true}) [2024-09-26 16:00:55.632048] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337655632040, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632056] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272832cc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3730526293242960203, dag_cnt=28, dag_type_cnts=27) [2024-09-26 16:00:55.632058] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272832cc0) [2024-09-26 16:00:55.632059] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337655632059, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405519, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101065}, checkpoint_unit_ptr:0x7f92a1a7db40}, is_freeze_clock:true}) [2024-09-26 16:00:55.632062] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337655632054, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632072] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f92728331f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=4663792628280492528, dag_cnt=29, dag_type_cnts=28) [2024-09-26 16:00:55.632073] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f92728331f0) [2024-09-26 16:00:55.632075] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337655632075, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405536, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:333}, checkpoint_unit_ptr:0x7f92a3bc4850}, is_freeze_clock:true}) [2024-09-26 16:00:55.632078] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337655632069, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632086] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272833720, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8399557287176188414, dag_cnt=30, dag_type_cnts=29) [2024-09-26 16:00:55.632088] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272833720) [2024-09-26 16:00:55.632089] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337655632089, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405552, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101066}, checkpoint_unit_ptr:0x7f92a3bc5020}, is_freeze_clock:true}) [2024-09-26 16:00:55.632092] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337655632084, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632101] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9272833c50, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8068938909778324349, dag_cnt=31, dag_type_cnts=30) [2024-09-26 16:00:55.632103] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272833c50) [2024-09-26 16:00:55.632104] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, create_flush_dag_time:1727337655632104, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405569, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:334}, checkpoint_unit_ptr:0x7f92a3bc57f0}, is_freeze_clock:true}) [2024-09-26 16:00:55.632107] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337655632099, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632115] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272834180, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7665372330263152188, dag_cnt=32, dag_type_cnts=31) [2024-09-26 16:00:55.632117] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272834180) [2024-09-26 16:00:55.632118] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, create_flush_dag_time:1727337655632118, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405586, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:451}, checkpoint_unit_ptr:0x7f92a3bc4080}, is_freeze_clock:true}) [2024-09-26 16:00:55.632121] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337655632113, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632130] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f92728346b0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6759548225246904317, dag_cnt=33, dag_type_cnts=32) [2024-09-26 16:00:55.632131] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f92728346b0) [2024-09-26 16:00:55.632133] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, create_flush_dag_time:1727337655632133, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405602, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:378}, checkpoint_unit_ptr:0x7f92a1a7f2b0}, is_freeze_clock:true}) [2024-09-26 16:00:55.632136] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337655632128, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632144] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272834be0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8944614441376604790, dag_cnt=34, dag_type_cnts=33) [2024-09-26 16:00:55.632145] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272834be0) [2024-09-26 16:00:55.632147] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, create_flush_dag_time:1727337655632147, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209456003, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:121}, checkpoint_unit_ptr:0x7f92a3bc5fc0}, is_freeze_clock:true}) [2024-09-26 16:00:55.632150] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337655632142, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632158] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272835110, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=507863347806783454, dag_cnt=35, dag_type_cnts=34) [2024-09-26 16:00:55.632160] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272835110) [2024-09-26 16:00:55.632161] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, create_flush_dag_time:1727337655632161, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405633, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:215}, checkpoint_unit_ptr:0x7f92a3bc86d0}, is_freeze_clock:true}) [2024-09-26 16:00:55.632164] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337655632156, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632172] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272835640, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1604282355669434429, dag_cnt=36, dag_type_cnts=35) [2024-09-26 16:00:55.632174] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272835640) [2024-09-26 16:00:55.632175] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, create_flush_dag_time:1727337655632175, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405649, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:456}, checkpoint_unit_ptr:0x7f92a3bc6790}, is_freeze_clock:true}) [2024-09-26 16:00:55.632178] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337655632170, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632187] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272835b70, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3553649416163308074, dag_cnt=37, dag_type_cnts=36) [2024-09-26 16:00:55.632188] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272835b70) [2024-09-26 16:00:55.632190] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337655632190, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405666, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:154}, checkpoint_unit_ptr:0x7f92a3bc6f60}, is_freeze_clock:true}) [2024-09-26 16:00:55.632193] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337655632185, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632201] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f92728360a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8097406237500329300, dag_cnt=38, dag_type_cnts=37) [2024-09-26 16:00:55.632203] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f92728360a0) [2024-09-26 16:00:55.632204] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337655632204, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405682, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101015}, checkpoint_unit_ptr:0x7f92a3bc7730}, is_freeze_clock:true}) [2024-09-26 16:00:55.632207] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337655632199, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632216] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f92728365d0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3976350754940852513, dag_cnt=39, dag_type_cnts=38) [2024-09-26 16:00:55.632218] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f92728365d0) [2024-09-26 16:00:55.632219] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337655632219, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405699, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101016}, checkpoint_unit_ptr:0x7f92a3bc7f00}, is_freeze_clock:true}) [2024-09-26 16:00:55.632222] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337655632214, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632231] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272836b00, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8720304653345806796, dag_cnt=40, dag_type_cnts=39) [2024-09-26 16:00:55.632233] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272836b00) [2024-09-26 16:00:55.632234] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, create_flush_dag_time:1727337655632234, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405716, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101063}, checkpoint_unit_ptr:0x7f92a3bca610}, is_freeze_clock:true}) [2024-09-26 16:00:55.632237] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337655632228, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632246] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272837030, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1830562711513853292, dag_cnt=41, dag_type_cnts=40) [2024-09-26 16:00:55.632247] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272837030) [2024-09-26 16:00:55.632249] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, create_flush_dag_time:1727337655632249, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405733, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:452}, checkpoint_unit_ptr:0x7f92a3bc9e40}, is_freeze_clock:true}) [2024-09-26 16:00:55.632252] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337655632243, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632260] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272837560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1536604874905918360, dag_cnt=42, dag_type_cnts=41) [2024-09-26 16:00:55.632262] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272837560) [2024-09-26 16:00:55.632263] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337655632263, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405750, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:328}, checkpoint_unit_ptr:0x7f92a3bc8ea0}, is_freeze_clock:true}) [2024-09-26 16:00:55.632267] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337655632258, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632278] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9272837a90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6933628521298738662, dag_cnt=43, dag_type_cnts=42) [2024-09-26 16:00:55.632280] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272837a90) [2024-09-26 16:00:55.632282] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337655632282, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405767, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:330}, checkpoint_unit_ptr:0x7f92a3bc9670}, is_freeze_clock:true}) [2024-09-26 16:00:55.632285] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337655632275, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632293] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9272837fc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1056199536334423564, dag_cnt=44, dag_type_cnts=43) [2024-09-26 16:00:55.632295] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9272837fc0) [2024-09-26 16:00:55.632296] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337655632296, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405784, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:329}, checkpoint_unit_ptr:0x7f92a3bcb5b0}, is_freeze_clock:true}) [2024-09-26 16:00:55.632299] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337655632291, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632328] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f92728384f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7405463607918801432, dag_cnt=45, dag_type_cnts=44) [2024-09-26 16:00:55.632332] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] success to create and add dag(ret=0, dag=0x7f92728384f0) [2024-09-26 16:00:55.632334] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, create_flush_dag_time:1727337655632334, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405801, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:332}, checkpoint_unit_ptr:0x7f92a3bcade0}, is_freeze_clock:true}) [2024-09-26 16:00:55.632337] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337655632306, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632347] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9272838a20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6699723950958743903, dag_cnt=46, dag_type_cnts=45) [2024-09-26 16:00:55.632349] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9272838a20) [2024-09-26 16:00:55.632350] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, create_flush_dag_time:1727337655632350, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405818, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101067}, checkpoint_unit_ptr:0x7f92a3bcbd80}, is_freeze_clock:true}) [2024-09-26 16:00:55.632353] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337655632345, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.632360] INFO [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:655) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] traversal_flush successfully(ls_->get_ls_id()={id:1}, flush_tasks=[{table:0x7f92a1a70850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a70080, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71020, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a717f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71fc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72790, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72f60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73730, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73f00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a746d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75670, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a74ea0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76610, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75e40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a775b0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76de0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a77d80, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7a490, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a79cc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a794f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78550, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78d20, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7ac60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7cba0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7b430, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7bc00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7c3d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7d370, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7e310, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7eae0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7db40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a3bc4850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}]) [2024-09-26 16:00:55.632415] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=52] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-09-26 16:00:55.632430] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337655632426, task_id:YB42C0A8002D-0006230056D8C6CE-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.632435] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=4] schedule one task(task=0x7f927282a3f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.632444] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337655632443, task_id:YB42C0A8002D-0006230056D8C6CF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.632447] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=2] schedule one task(task=0x7f927282a920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.632468] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] succeed to add sys task(task={start_time:1727337655632467, task_id:YB42C0A8002D-0006230056D8C6D0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.632471] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=2] schedule one task(task=0x7f927282ae50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.632474] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655632474, task_id:YB42C0A8002D-0006230056D8C6D1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.632476] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=2] schedule one task(task=0x7f927282b380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.632481] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655632480, task_id:YB42C0A8002D-0006230056D8C6D2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.632483] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=2] schedule one task(task=0x7f927282b8b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.632487] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655632486, task_id:YB42C0A8002D-0006230056D8C6D3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.632489] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=1] schedule one task(task=0x7f927282bde0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.632508] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.632514] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.632515] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.632518] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.632518] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726157244792838000) [2024-09-26 16:00:55.632521] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.632522] INFO [STORAGE] ready_for_flush (ob_tx_data_memtable.cpp:865) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=3] memtable is frozen yet.(this=0x7f92a2294080) [2024-09-26 16:00:55.632529] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.632536] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:00:55.632547] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241777443612000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.632559] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.632562] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.632563] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:55.632567] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.632567] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.632569] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.632571] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:55.632571] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:55.632574] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.632581] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.632603] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.632605] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.632607] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:00:55.632610] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.632612] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847222}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:00:55.632617] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.632630] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=30][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.632632] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.632634] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:00:55.632637] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.640506] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.640525] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=18] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337655631649, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:55.640541] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.640580] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:55.640594] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=8] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.640634] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=13] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:55.640640] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.640649] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847222}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:55.640675] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=18] succeed to build merge ctx(tablet_id={id:1152921504606847222}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847222}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655632599, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=8.05ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.640927] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=26] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:00:55.641668] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=33] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=8366, total_rows=58562) [2024-09-26 16:00:55.642111] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=27] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D1-0-0, dag_ret:0, dag_status:2, start_time:1727337655632476, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db7470, output_block_cnt_arr:0x7f9287db7478, concurrent_cnt:1, estimate_row_cnt:329211, estimate_occupy_size:150994944, latest_update_ts:1727337655632599, estimated_finish_time:1727337672642182}) [2024-09-26 16:00:55.642125] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=14] succeed to init merge ctx(task={this:0x7f927282b380, type:15, status:2, dag:{ObIDag:{this:0x7f927282afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D1-0-0, dag_ret:0, dag_status:2, start_time:1727337655632476, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.642143] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=17] task finish process(ret=0, start_time=1727337655632598, end_time=1727337655642142, runtime=9544, *this={this:0x7f927282b380, type:15, status:2, dag:{ObIDag:{this:0x7f927282afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D1-0-0, dag_ret:0, dag_status:2, start_time:1727337655632476, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.642171] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=3] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.642250] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a68060, count:4, allocator:0x7f926361c7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a68350, local_datums:0x7f9264a68350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:55.642283] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=29] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d38060, count:4, allocator:0x7f926361d558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d38350, local_datums:0x7f9287d38350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:55.642529] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=9][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:55.642590] ERROR issue_dba_error (ob_log.cpp:1875) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.642595] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.642603] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:55.642606] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:55.642608] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9287dd8140, hex: 3230323430393131303133333438, cstr: 20240911013348},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9287dd814e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd80e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd8118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3080, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7221, data_checksum:4284205003, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265b00030, buf_size:7221, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1004563202, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:00:55.642624] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=16][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:55.642627] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=3][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:55.642628] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:55.642630] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:55.642632] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a68350, local_datums:0x7f9264a68350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a683d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a68350, local_datums:0x7f9264a68350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a683d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f926361c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926361c118, limit_param:null, stmt_allocator:0x7f926361c7d8, allocator:0x7f926361c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a68350, local_datums:0x7f9264a68350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a683d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:00:55.642693] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=61][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:55.642695] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a68350, local_datums:0x7f9264a68350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a683d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f926361c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f926361c118, limit_param:null, stmt_allocator:0x7f926361c7d8, allocator:0x7f926361c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a68350, local_datums:0x7f9264a68350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a683d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a68408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:00:55.642743] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=48][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:55.642746] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=2] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.642760] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=10] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337655631649, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.642770] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:55.642785] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=3][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:55.642789] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:55.642791] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337655642196, end_time=1727337655642791, runtime=595, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f927282afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D1-0-0, dag_ret:0, dag_status:2, start_time:1727337655632476, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.642799] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f927282afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D1-0-0, dag_ret:0, dag_status:2, start_time:1727337655632476, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:55.642808] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=10331, dag_cnt=45, dag_cnts_[dag.get_type()]=44, &dag=0x7f927282afc0, dag={ObIDag:{this:0x7f927282afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D1-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655632476, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.642819] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D1-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337655632474, task_id:YB42C0A8002D-0006230056D8C6D1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.642839] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337655642836, task_id:YB42C0A8002D-0006230056D8C6D4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.642846] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=6] schedule one task(task=0x7f927282c310, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.642882] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.642886] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.642888] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.642894] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.642944] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.642954] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=10] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337655631664, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:55.642961] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.643049] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:55.643053] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.643059] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847223}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:55.643078] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:1152921504606847223}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847223}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655632554, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=10.50ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.643326] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=23] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:00:55.644047] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=7999, total_rows=55993) [2024-09-26 16:00:55.644438] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D2-0-0, dag_ret:0, dag_status:2, start_time:1727337655632483, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515470, output_block_cnt_arr:0x7f9268515478, concurrent_cnt:1, estimate_row_cnt:332829, estimate_occupy_size:150994944, latest_update_ts:1727337655632554, estimated_finish_time:1727337672644510}) [2024-09-26 16:00:55.644468] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=30] succeed to init merge ctx(task={this:0x7f927282b8b0, type:15, status:2, dag:{ObIDag:{this:0x7f927282b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D2-0-0, dag_ret:0, dag_status:2, start_time:1727337655632483, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.644474] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=6] task finish process(ret=0, start_time=1727337655632553, end_time=1727337655644474, runtime=11921, *this={this:0x7f927282b8b0, type:15, status:2, dag:{ObIDag:{this:0x7f927282b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D2-0-0, dag_ret:0, dag_status:2, start_time:1727337655632483, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.644493] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=2] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.644523] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726158220122116000) [2024-09-26 16:00:55.644534] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.644527] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337655631724, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.644542] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=14] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752755184, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828752755184, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.644558] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.644562] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.644569] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:226}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752755184, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.644590] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:226}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752755184, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:226}, table_count:1, [{i:0, table_key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}, merge_scn:{val:1726241777443612000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655632508, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=12.06ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.644597] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:4, allocator:0x7f92865547d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665f8350, local_datums:0x7f92665f8350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:55.644615] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=15] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c72060, count:4, allocator:0x7f9286555558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9262c72350, local_datums:0x7f9262c72350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:55.644618] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=22] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D3-0-0, dag_ret:0, dag_status:2, start_time:1727337655632489, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5480, output_block_cnt_arr:0x7f9287db5488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:56623104, latest_update_ts:1727337655632508, estimated_finish_time:1727337672644645}) [2024-09-26 16:00:55.644625] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f927282bde0, type:15, status:2, dag:{ObIDag:{this:0x7f927282ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D3-0-0, dag_ret:0, dag_status:2, start_time:1727337655632489, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:55.644631] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=6] task finish process(ret=0, start_time=1727337655632505, end_time=1727337655644631, runtime=12126, *this={this:0x7f927282bde0, type:15, status:2, dag:{ObIDag:{this:0x7f927282ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D3-0-0, dag_ret:0, dag_status:2, start_time:1727337655632489, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:55.644648] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=12] schedule one task(task=0x7f9264afa030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.644806] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.644819] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=12] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337655631632, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=17, max_schema_version_on_memtable=1725120002010696) [2024-09-26 16:00:55.644826] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=6] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=17, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.644863] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=7][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:55.644915] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.644918] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=6] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1887, total_rows=92463) [2024-09-26 16:00:55.644921] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.644919] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.644926] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:55.644928] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:55.644927] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200489}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:19, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.644930] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9262c76140, hex: 3230323430393131303133333131, cstr: 20240911013311},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9262c7614e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c760e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-15038, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7227, data_checksum:1452123993, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92665a6030, buf_size:7227, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1818264584, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:00:55.644944] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=14][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:55.644946] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:55.644947] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:55.644949] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:55.644943] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:200489}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200489}, table_count:2, [{i:0, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655632564, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=12.36ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.644950] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665f8350, local_datums:0x7f92665f8350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f83d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665f8350, local_datums:0x7f92665f8350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f83d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9286554118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286554118, limit_param:null, stmt_allocator:0x7f92865547d8, allocator:0x7f92865547d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665f8350, local_datums:0x7f92665f8350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f83d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:00:55.645006] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=56][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:55.645007] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665f8350, local_datums:0x7f92665f8350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f83d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9286554118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286554118, limit_param:null, stmt_allocator:0x7f92865547d8, allocator:0x7f92865547d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92665f8350, local_datums:0x7f92665f8350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f83d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665f8408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:00:55.645046] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=31][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:55.645050] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=3] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.645062] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=9] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337655631664, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.645080] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=15][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:55.645092] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:55.645095] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:55.645097] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337655644550, end_time=1727337655645097, runtime=547, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f927282b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D2-0-0, dag_ret:0, dag_status:2, start_time:1727337655632483, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.645104] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f927282b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D2-0-0, dag_ret:0, dag_status:2, start_time:1727337655632483, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:55.645112] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=12629, dag_cnt=44, dag_cnts_[dag.get_type()]=43, &dag=0x7f927282b4f0, dag={ObIDag:{this:0x7f927282b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D2-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655632483, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.645122] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D2-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655632480, task_id:YB42C0A8002D-0006230056D8C6D2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.645202] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=226, range=0x7f92aaad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:5, allocator:0x7f92bc8f47d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3e360, local_datums:0x7f9263a3e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:55.645258] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D0-0-0, dag_ret:0, dag_status:2, start_time:1727337655632471, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab460, output_block_cnt_arr:0x7f92684ab468, concurrent_cnt:1, estimate_row_cnt:254187, estimate_occupy_size:138412032, latest_update_ts:1727337655632564, estimated_finish_time:1727337672645324}) [2024-09-26 16:00:55.645269] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=10] succeed to init merge ctx(task={this:0x7f927282ae50, type:15, status:2, dag:{ObIDag:{this:0x7f927282aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D0-0-0, dag_ret:0, dag_status:2, start_time:1727337655632471, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.645274] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=5] task finish process(ret=0, start_time=1727337655632562, end_time=1727337655645274, runtime=12712, *this={this:0x7f927282ae50, type:15, status:2, dag:{ObIDag:{this:0x7f927282aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D0-0-0, dag_ret:0, dag_status:2, start_time:1727337655632471, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.645296] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=4] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.645305] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337655645303, task_id:YB42C0A8002D-0006230056D8C6D5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.645321] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=15] schedule one task(task=0x7f927282c840, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.645898] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:55.645903] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.651607] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=13][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:55.651667] ERROR issue_dba_error (ob_log.cpp:1875) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=18][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.651672] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.651680] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:55.651685] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=5][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:55.651687] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d58070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d580a8, hex: 0000000000000000, int: 0},idx=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9264d580e0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d58118, hex: 58A3C56A4A280BE8, int: -1726241732082425000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d58150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-19590, column_count:10, rowkey_column_count:5, has_column_checksum:0, row_count:224, row_store_type:0, opt:0, var_column_count:0, row_offset:15527, original_length:16363, max_merged_trans_version:1726241777443612000, data_length:16363, data_zlength:16363, data_checksum:2057046089, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6a04090, buf_size:16363, data_size:16363, row_count:224, column_count:10, max_merged_trans_version:1726241777443612000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1006468742, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16363}) [2024-09-26 16:00:55.651715] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=27][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:55.651717] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:55.651719] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:55.651720] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:55.651722] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:55.651723] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:55.651725] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3e360, local_datums:0x7f9263a3e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3e360, local_datums:0x7f9263a3e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f92bc8f4118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92bc8f4118, limit_param:null, stmt_allocator:0x7f92bc8f47d8, allocator:0x7f92bc8f47d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3e360, local_datums:0x7f9263a3e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337655631724, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}) [2024-09-26 16:00:55.651792] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=66][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:55.651794] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3e360, local_datums:0x7f9263a3e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f92bc8f4118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92bc8f4118, limit_param:null, stmt_allocator:0x7f92bc8f47d8, allocator:0x7f92bc8f47d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3e360, local_datums:0x7f9263a3e360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3e3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3e450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337655631724, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}]) [2024-09-26 16:00:55.651832] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=37][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:55.651834] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=1] partition merge iter row count(i=0, row_count=224, ghost_row_count=0, pkey={tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, table={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337655631724, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.651847] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:55.651857] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:55.651861] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:55.651863] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337655645158, end_time=1727337655651863, runtime=6705, *this={this:0x7f9264afa030, type:1, status:2, dag:{ObIDag:{this:0x7f927282ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D3-0-0, dag_ret:0, dag_status:2, start_time:1727337655632489, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:00:55.651869] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264afa030, type:1, status:2, dag:{ObIDag:{this:0x7f927282ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D3-0-0, dag_ret:0, dag_status:2, start_time:1727337655632489, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:55.651887] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=19398, dag_cnt=43, dag_cnts_[dag.get_type()]=42, &dag=0x7f927282ba20, dag={ObIDag:{this:0x7f927282ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D3-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655632489, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}) [2024-09-26 16:00:55.651898] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D3-0-0] [lt=8] succeed to del sys task(removed_task={start_time:1727337655632486, task_id:YB42C0A8002D-0006230056D8C6D3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.651927] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655651923, task_id:YB42C0A8002D-0006230056D8C6D6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.651934] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=6] schedule one task(task=0x7f927282cdf0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.651958] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.651963] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.651966] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.651972] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.652034] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:3, allocator:0x7f92864e07d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9263a38320, local_datums:0x7f9263a38320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:55.652061] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=23] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5c060, count:3, allocator:0x7f92864e1558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264d5c320, local_datums:0x7f9264d5c320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:00:55.652206] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.652215] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=9][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.652225] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=10][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.652231] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.652232] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=10][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:55.652236] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.652246] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.652275] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.652279] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.652294] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=15][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:55.652297] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:55.652299] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92729f2070, hex: 9F094D0300000000, int: 55380383},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92729f20a8, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92729f20e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3044, column_count:19, rowkey_column_count:3, has_column_checksum:0, row_count:83, row_store_type:0, opt:5, var_column_count:0, row_offset:16109, original_length:16381, max_merged_trans_version:1726163995623997000, data_length:16381, data_zlength:7759, data_checksum:3416849541, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9264dce030, buf_size:7759, data_size:16381, row_count:83, column_count:19, max_merged_trans_version:1726163995623997000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3537734793, row_count_delta:83, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16381}) [2024-09-26 16:00:55.652312] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=12][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:55.652313] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:55.652315] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:55.652317] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:55.652319] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264d5c320, local_datums:0x7f9264d5c320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264d5c4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264d5c6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264d5c320, local_datums:0x7f9264d5c320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264d5c4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264d5c6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f92864e0e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92864e0e98, limit_param:null, stmt_allocator:0x7f92864e1558, allocator:0x7f92864e1558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320133, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264d5c320, local_datums:0x7f9264d5c320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264d5c4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264d5c6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337655631632, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}) [2024-09-26 16:00:55.652395] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=75][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:55.652397] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264d5c320, local_datums:0x7f9264d5c320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264d5c4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264d5c6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f92864e0e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92864e0e98, limit_param:null, stmt_allocator:0x7f92864e1558, allocator:0x7f92864e1558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320133, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264d5c320, local_datums:0x7f9264d5c320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264d5c4f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5c5d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264d5c6e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337655631632, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}]) [2024-09-26 16:00:55.652435] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=38][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:55.652437] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=2] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.652448] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=9] partition merge iter row count(i=1, row_count=83, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, table={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337655631632, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.652457] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:55.652466] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:55.652470] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:55.652471] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337655651992, end_time=1727337655652471, runtime=479, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f927282aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D0-0-0, dag_ret:0, dag_status:2, start_time:1727337655632471, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.652479] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f927282aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D0-0-0, dag_ret:0, dag_status:2, start_time:1727337655632471, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:55.652487] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=20015, dag_cnt=42, dag_cnts_[dag.get_type()]=41, &dag=0x7f927282aa90, dag={ObIDag:{this:0x7f927282aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D0-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655632471, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.652495] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D0-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655632467, task_id:YB42C0A8002D-0006230056D8C6D0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.652510] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337655652509, task_id:YB42C0A8002D-0006230056D8C6D7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.652528] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=17] schedule one task(task=0x7f927282d320, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.652552] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.652556] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.652558] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.652563] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.652567] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.652575] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241621541056000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.652638] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b863bfe0, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:506, tablet_addr:[816017-1849-0-2305843647017062400-1](offset=724992,size=594,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, data_tablet_id:{id:200489}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725991206240736000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1725991205062061000, cur_report_version:1725991205062061000, data_checksum:2143506547, row_count:46137450}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1725120000848352, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1725120000848352}, table_store_addr:{ptr:null, addr:[816017-1849-0-2305845196426506240-1](offset=716800,size=2037,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718137333-0](offset=718837,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255101290641-0](offset=718993,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1725991205062061000}, medium_info_list:{ptr:(nil), addr:[816017-1849-0-2305843520315523072-0](offset=720896,size=476,type=2,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1725120001569133000, v:0}, create_commit_version:1725120001569133000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a746d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:55.652674] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=36] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:55.652680] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.652736] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.652747] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=10] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337655631601, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=18, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:00:55.652754] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=6] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.652955] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:55.652963] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=7] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.652970] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200240}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.653007] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:200240}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200240}, table_count:2, [{i:0, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8637b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655632628, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=20.34ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.653225] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=22] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:00:55.653758] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=6] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CF-0-0, dag_ret:0, dag_status:2, start_time:1727337655632446, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1460, output_block_cnt_arr:0x7f92728f1468, concurrent_cnt:1, estimate_row_cnt:85417, estimate_occupy_size:134217728, latest_update_ts:1727337655632628, estimated_finish_time:1727337672653821}) [2024-09-26 16:00:55.653773] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=15] succeed to init merge ctx(task={this:0x7f927282a920, type:15, status:2, dag:{ObIDag:{this:0x7f927282a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CF-0-0, dag_ret:0, dag_status:2, start_time:1727337655632446, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.653780] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=6] task finish process(ret=0, start_time=1727337655632627, end_time=1727337655653779, runtime=21152, *this={this:0x7f927282a920, type:15, status:2, dag:{ObIDag:{this:0x7f927282a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CF-0-0, dag_ret:0, dag_status:2, start_time:1727337655632446, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.653798] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=2] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.653866] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:3, allocator:0x7f92636287d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:55.653886] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=17] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:3, allocator:0x7f9263629558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5e320, local_datums:0x7f9263b5e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:55.654097] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=9][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:55.654142] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.654146] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf2041c8 0xf217c49 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.654153] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:55.654155] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:55.654157] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9264d62108, hex: 313030303030333830, cstr: 100000380},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d620a8, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d620e0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-29413, column_count:20, rowkey_column_count:3, has_column_checksum:0, row_count:100, row_store_type:0, opt:0, var_column_count:0, row_offset:16082, original_length:16422, max_merged_trans_version:1726189672100745000, data_length:16454, data_zlength:5272, data_checksum:4171874375, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9263bde030, buf_size:5272, data_size:16454, row_count:100, column_count:20, max_merged_trans_version:1726189672100745000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:371002229, row_count_delta:22, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16422}) [2024-09-26 16:00:55.654169] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=12][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:55.654171] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:55.654175] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=4][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:55.654177] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:55.654179] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1526) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, partition_fuser_->get_result_row()={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b58148, local_datums:0x7f9263b58148}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b58190, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b581c8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b583c0, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b583f8, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, fuse_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9263628118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263628118, limit_param:null, stmt_allocator:0x7f92636287d8, allocator:0x7f92636287d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5e320, local_datums:0x7f9263b5e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9263628e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263628e98, limit_param:null, stmt_allocator:0x7f9263629558, allocator:0x7f9263629558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5e320, local_datums:0x7f9263b5e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337655631601, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:55.654282] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=103][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9263628118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263628118, limit_param:null, stmt_allocator:0x7f92636287d8, allocator:0x7f92636287d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5a320, local_datums:0x7f9263b5a320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5e320, local_datums:0x7f9263b5e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f9263628e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263628e98, limit_param:null, stmt_allocator:0x7f9263629558, allocator:0x7f9263629558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5e320, local_datums:0x7f9263b5e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337655631601, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:00:55.654354] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=71][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:55.654356] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=1] partition merge iter row count(i=0, row_count=23, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.654367] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=8] partition merge iter row count(i=1, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337655631601, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.654375] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:55.654384] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:55.654388] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:55.654389] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337655653819, end_time=1727337655654389, runtime=570, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f927282a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CF-0-0, dag_ret:0, dag_status:2, start_time:1727337655632446, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.654396] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f927282a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CF-0-0, dag_ret:0, dag_status:2, start_time:1727337655632446, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:55.654421] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=21] dag finished(dag_ret=-4184, runtime=21975, dag_cnt=41, dag_cnts_[dag.get_type()]=40, &dag=0x7f927282a560, dag={ObIDag:{this:0x7f927282a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6CF-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655632446, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.654429] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6CF-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655632443, task_id:YB42C0A8002D-0006230056D8C6CF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.654445] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337655654444, task_id:YB42C0A8002D-0006230056D8C6D8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.654449] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=4] schedule one task(task=0x7f927282d850, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.654514] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1, storage_schema:{this:0x7f9263a4b2b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.654526] INFO [STORAGE] prepare_tx_data_list (ob_tx_data_memtable.cpp:266) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=10] construct list more than once. skip this time(this=0x7f92a2294080) [2024-09-26 16:00:55.654599] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.654605] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.654607] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.654613] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.654617] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.654624] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.654645] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337655631776, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.654658] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828761605424, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828761605424, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.654672] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.654675] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.654681] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200004}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1710828761605424, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.654699] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:200004}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828761605424, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200004}, table_count:1, [{i:0, table_key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8621da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655652199, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.48ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.654723] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282ca30, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D6-0-0, dag_ret:0, dag_status:2, start_time:1727337655651934, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5460, output_block_cnt_arr:0x7f9287db5468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337655652199, estimated_finish_time:1727337672654726}) [2024-09-26 16:00:55.654731] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f927282cdf0, type:15, status:2, dag:{ObIDag:{this:0x7f927282ca30, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D6-0-0, dag_ret:0, dag_status:2, start_time:1727337655651934, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.654737] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=6] task finish process(ret=0, start_time=1727337655652196, end_time=1727337655654737, runtime=2541, *this={this:0x7f927282cdf0, type:15, status:2, dag:{ObIDag:{this:0x7f927282ca30, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D6-0-0, dag_ret:0, dag_status:2, start_time:1727337655651934, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.654754] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=2] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.654820] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=200004, range=0x7f92ab056200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:3, allocator:0x7f92636287d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664f0320, local_datums:0x7f92664f0320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:55.654987] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=25][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:55.655033] ERROR issue_dba_error (ob_log.cpp:1875) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.655037] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.655041] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=4][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:55.655043] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:55.655045] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc0a8, hex: C852A9F488570BE8, int: -1726189786434547000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc0e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19071, column_count:8, rowkey_column_count:3, has_column_checksum:0, row_count:218, row_store_type:0, opt:0, var_column_count:0, row_offset:15549, original_length:16361, max_merged_trans_version:1726190661653507000, data_length:16361, data_zlength:5996, data_checksum:2604580500, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9262c6e030, buf_size:5996, data_size:16361, row_count:218, column_count:8, max_merged_trans_version:1726190661653507000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:83488362, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16361}) [2024-09-26 16:00:55.655056] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:55.655058] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:55.655059] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:55.655061] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:55.655062] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664f0320, local_datums:0x7f92664f0320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664f0320, local_datums:0x7f92664f0320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f9263628118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263628118, limit_param:null, stmt_allocator:0x7f92636287d8, allocator:0x7f92636287d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:34477281, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664f0320, local_datums:0x7f92664f0320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337655631776, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:55.655124] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=61][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:55.655140] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=15][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664f0320, local_datums:0x7f92664f0320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f9263628118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263628118, limit_param:null, stmt_allocator:0x7f92636287d8, allocator:0x7f92636287d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:34477281, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664f0320, local_datums:0x7f92664f0320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337655631776, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:55.655179] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=39][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:55.655181] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=1] partition merge iter row count(i=0, row_count=218, ghost_row_count=0, pkey={tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337655631776, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.655193] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:55.655204] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:55.655208] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:55.655209] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337655654769, end_time=1727337655655209, runtime=440, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f927282ca30, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D6-0-0, dag_ret:0, dag_status:2, start_time:1727337655651934, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.655216] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f927282ca30, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D6-0-0, dag_ret:0, dag_status:2, start_time:1727337655651934, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:55.655224] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=3290, dag_cnt=40, dag_cnts_[dag.get_type()]=39, &dag=0x7f927282ca30, dag={ObIDag:{this:0x7f927282ca30, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D6-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655651934, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.655232] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D6-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655651923, task_id:YB42C0A8002D-0006230056D8C6D6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.655260] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337655655259, task_id:YB42C0A8002D-0006230056D8C6D9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.655264] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=3] schedule one task(task=0x7f927282dd80, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.655371] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.655376] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.655379] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.655384] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.655388] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.655395] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.658543] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337655631822, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.658560] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=17] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753327496, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753327496, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.658578] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=13] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.658582] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.658589] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:444}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753327496, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.658610] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:444}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753327496, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:444}, table_count:1, [{i:0, table_key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8632d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655655366, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.22ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.658638] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282d9c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D9-0-0, dag_ret:0, dag_status:2, start_time:1727337655655264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515490, output_block_cnt_arr:0x7f9268515498, concurrent_cnt:1, estimate_row_cnt:4, estimate_occupy_size:6291456, latest_update_ts:1727337655655366, estimated_finish_time:1727337672658641}) [2024-09-26 16:00:55.658646] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f927282dd80, type:15, status:2, dag:{ObIDag:{this:0x7f927282d9c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D9-0-0, dag_ret:0, dag_status:2, start_time:1727337655655264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.658652] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=6] task finish process(ret=0, start_time=1727337655655363, end_time=1727337655658652, runtime=3289, *this={this:0x7f927282dd80, type:15, status:2, dag:{ObIDag:{this:0x7f927282d9c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D9-0-0, dag_ret:0, dag_status:2, start_time:1727337655655264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.658696] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=4] schedule one task(task=0x7f9272838f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.659525] INFO [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:329) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=2] pre-process commit versions row(get_ls_id()={id:1}, do_recycle_=true, current_time=1727337655659524, prev_recycle_time=0) [2024-09-26 16:00:55.661665] INFO [STORAGE] get_ls_min_end_scn (ob_ls_tablet_service.cpp:5618) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=6] get ls min end scn finish(ls_checkpoint={val:1726163586617842000, v:0}) [2024-09-26 16:00:55.661695] WDIAG [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:761) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=24][errcode=-4023] failed to get snapshot version(ret=-4023, MTL_ID()=1) [2024-09-26 16:00:55.661700] INFO [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:773) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=4] get tx data recycle scn finish.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, recycle_scn={val:1726076389100953000, v:0}, min_end_scn_from_old_tablets={val:4611686018427387903, v:0}, min_end_scn_from_latest_tablets={val:1726076389100953000, v:0}) [2024-09-26 16:00:55.661707] WDIAG [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:336) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=6][errcode=-4023] get recycle ts failed.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:55.661713] WDIAG [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:236) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=2][errcode=-4023] process commit version row failed.(ret=-4023, ret="OB_EAGAIN", this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337655631535, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:55.661730] INFO [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:254) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=17] [TX DATA MERGE]pre process for merge done(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}, this=0x7f92a2294080, tg=time guard have no click events for optimization, fake_tx_data_guard={tx_data:{tx_id:{txid:9223372036854775807}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337655631535, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:55.661745] WDIAG [STORAGE.COMPACTION] pre_process_tx_data_table_merge_ (ob_tx_table_merge_task.cpp:117) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=14][errcode=-4023] do pre process for tx data table merge failed.(ret=-4023, ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, table={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337655631535, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:00:55.661757] WDIAG [STORAGE.COMPACTION] inner_init_ctx (ob_tx_table_merge_task.cpp:157) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=11][errcode=-4023] pre process tx data table for merge failed.(ret=-4023, ret="OB_EAGAIN", ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.661762] WDIAG [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1061) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=4][errcode=-4023] fail to inner init ctx(ret=-4023, tablet_id={id:49402}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, create_snapshot_version:0, is_full_merge:false, merge_level:1, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:4, range_array:[], concurrent_cnt:0, is_inited:false}, schema_ctx:{schema_version:1, storage_schema:{this:0x7f9263a4b2b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:49402}, table_count:1, [{i:0, table_key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref:4}]}, schedule_major:false, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}, merge_scn:{val:4611686018427387903, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:NULL, compaction_filter:NULL, time_guard:total=29.25ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}) [2024-09-26 16:00:55.661797] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:975) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=35][errcode=-4023] failed to build merge ctx(ret=-4023, ctx->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.661802] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:994) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=5][errcode=-4023] sstable merge finish(ret=-4023, ctx=0x7f9263a4a060, task={this:0x7f927282a3f0, type:15, status:2, dag:{ObIDag:{this:0x7f927282a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C6CE-0-0, dag_ret:0, dag_status:2, start_time:1727337655632435, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:55.661813] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=11][errcode=-4023] failed to process task(ret=-4023) [2024-09-26 16:00:55.661816] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=3] task finish process(ret=-4023, start_time=1727337655632499, end_time=1727337655661816, runtime=29317, *this={this:0x7f927282a3f0, type:15, status:2, dag:{ObIDag:{this:0x7f927282a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C6CE-0-0, dag_ret:0, dag_status:2, start_time:1727337655632435, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:00:55.661824] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=8][errcode=-4023] failed to do work(ret=-4023, *task_={this:0x7f927282a3f0, type:15, status:2, dag:{ObIDag:{this:0x7f927282a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C6CE-0-0, dag_ret:0, dag_status:2, start_time:1727337655632435, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:55.661834] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=7] dag finished(dag_ret=-4023, runtime=29399, dag_cnt=39, dag_cnts_[dag.get_type()]=0, &dag=0x7f927282a030, dag={ObIDag:{this:0x7f927282a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C6CE-0-0, dag_ret:-4023, dag_status:5, start_time:1727337655632435, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}) [2024-09-26 16:00:55.661845] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2849][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6CE-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655632426, task_id:YB42C0A8002D-0006230056D8C6CE-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.661861] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] succeed to add sys task(task={start_time:1727337655661857, task_id:YB42C0A8002D-0006230056D8C6DA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.661869] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=6] schedule one task(task=0x7f927282e2b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.661961] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=444, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:6, allocator:0x7f92a339c7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:55.662154] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=25][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:55.662208] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.662206] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.662214] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.662212] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.662219] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.662220] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:55.662223] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:55.662227] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.662226] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe0a8, hex: 0300000000000000, int: 3},idx=2:{len: 12, flag: 0, null: 0, ptr: 0x7f92665fe1b0, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe118, hex: 420B000000000000, int: 2882},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe150, hex: 288E681FE7280BE8, int: -1726241059036951000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe188, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:24981, column_count:9, rowkey_column_count:6, has_column_checksum:0, row_count:207, row_store_type:0, opt:0, var_column_count:0, row_offset:15610, original_length:16378, max_merged_trans_version:1726241059036951000, data_length:16378, data_zlength:16378, data_checksum:1574137226, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926f604090, buf_size:16378, data_size:16378, row_count:207, column_count:9, max_merged_trans_version:1726241059036951000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:666643265, row_count_delta:3, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16378}) [2024-09-26 16:00:55.662241] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=14][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:55.662243] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:55.662245] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:55.662255] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=10][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:55.662258] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=3][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:55.662260] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:55.662235] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=4] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8631da0, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:515, tablet_addr:[816017-1849-0-2305843656681984000-2](offset=1970176,size=603,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:309}, data_tablet_id:{id:309}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:810117565, row_count:8}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86324c4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844526412836864-0](offset=1945600,size=1413,type=2,seq=0), type:"MAJOR", tablet_id:{id:309}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077442448649001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843990615668101-0](offset=1947013,size=914,type=2,seq=0), type:"MINOR", tablet_id:{id:309}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178572306341000, upper_trans_version:1726178572306341000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337655631836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcdcc0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241316899724000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405581}, this:0x7f92a3bcdcc0, timestamp:1727334209405581, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:51, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277651138194000, v:0}, rec_scn:{val:1726241616921722000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697937408-2](offset=1957888,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843189604278604-0](offset=1958220,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881304564-0](offset=1958388,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447824598-0](offset=1958614,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a76de0, 0x7f92a3bcdcc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.662306] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=70] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.662318] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241316899724000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.662262] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f04b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f04b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f92a339c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a339c118, limit_param:null, stmt_allocator:0x7f92a339c7d8, allocator:0x7f92a339c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320133, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f04b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337655631822, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}) [2024-09-26 16:00:55.662329] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=67][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:55.662331] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f04b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f92a339c118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a339c118, limit_param:null, stmt_allocator:0x7f92a339c7d8, allocator:0x7f92a339c7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:4282320133, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f04b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337655631822, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}]) [2024-09-26 16:00:55.662368] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=37][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:55.662370] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=1] partition merge iter row count(i=0, row_count=207, ghost_row_count=0, pkey={tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337655631822, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.662386] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=11][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:55.662397] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:55.662401] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:55.662403] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337655661898, end_time=1727337655662403, runtime=505, *this={this:0x7f9272838f50, type:1, status:2, dag:{ObIDag:{this:0x7f927282d9c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D9-0-0, dag_ret:0, dag_status:2, start_time:1727337655655264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.662420] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=16][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9272838f50, type:1, status:2, dag:{ObIDag:{this:0x7f927282d9c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D9-0-0, dag_ret:0, dag_status:2, start_time:1727337655655264, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:55.662429] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=7164, dag_cnt=38, dag_cnts_[dag.get_type()]=38, &dag=0x7f927282d9c0, dag={ObIDag:{this:0x7f927282d9c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D9-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655655264, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.662437] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D9-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655655259, task_id:YB42C0A8002D-0006230056D8C6D9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.662457] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337655662454, task_id:YB42C0A8002D-0006230056D8C6DB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.662477] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=19] schedule one task(task=0x7f927282e7e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.662482] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337655631791, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.662500] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=18] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:0, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.662515] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.662518] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.662525] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:0, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.662544] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:1}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:0, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1}, table_count:1, [{i:0, table_key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}, merge_scn:{val:1726241621541056000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655652548, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=9.97ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.662571] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282cf60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D7-0-0, dag_ret:0, dag_status:2, start_time:1727337655652527, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}, scanned_row_cnt_arr:0x7f9287db7480, output_block_cnt_arr:0x7f9287db7488, concurrent_cnt:1, estimate_row_cnt:3, estimate_occupy_size:6291456, latest_update_ts:1727337655652548, estimated_finish_time:1727337672662573}) [2024-09-26 16:00:55.662577] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f927282d320, type:15, status:2, dag:{ObIDag:{this:0x7f927282cf60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D7-0-0, dag_ret:0, dag_status:2, start_time:1727337655652527, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:55.662582] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=5] task finish process(ret=0, start_time=1727337655652546, end_time=1727337655662582, runtime=10036, *this={this:0x7f927282d320, type:15, status:2, dag:{ObIDag:{this:0x7f927282cf60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D7-0-0, dag_ret:0, dag_status:2, start_time:1727337655652527, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:55.662599] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=4] schedule one task(task=0x7f9264afa030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.662609] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.662613] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.662617] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.662622] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.662653] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=29] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.662663] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241465393532000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.662723] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:00:55.662730] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.662769] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337655631739, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.662793] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=10] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:55.662798] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=5] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9263a43480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.662912] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3214, total_rows=22498) [2024-09-26 16:00:55.662915] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.662922] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847408}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.662939] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:1152921504606847408}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9263a43480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847408}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655642873, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=20.05ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.662942] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=1, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:8}, cols_desc:{ObIArray:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ec060, count:5, allocator:0x7f92865a27d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:55.663049] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D4-0-0, dag_ret:0, dag_status:2, start_time:1727337655642845, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a435d0, output_block_cnt_arr:0x7f9263a435d8, concurrent_cnt:1, estimate_row_cnt:35098, estimate_occupy_size:18874368, latest_update_ts:1727337655642873, estimated_finish_time:1727337672663057}) [2024-09-26 16:00:55.663056] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f927282c310, type:15, status:2, dag:{ObIDag:{this:0x7f927282bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D4-0-0, dag_ret:0, dag_status:2, start_time:1727337655642845, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.663062] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=5] task finish process(ret=0, start_time=1727337655642870, end_time=1727337655663062, runtime=20192, *this={this:0x7f927282c310, type:15, status:2, dag:{ObIDag:{this:0x7f927282bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D4-0-0, dag_ret:0, dag_status:2, start_time:1727337655642845, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.663087] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=4] schedule one task(task=0x7f9272926030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.663127] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:55.663467] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=24][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:55.663537] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=10][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.663543] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.663554] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=11][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:55.663559] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=4][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:55.663562] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92664f0178, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f00a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92664f0189, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0118, hex: 60D2640CCC290BE8, int: -1726240075808452000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-20032, column_count:8, rowkey_column_count:5, has_column_checksum:0, row_count:158, row_store_type:0, opt:0, var_column_count:0, row_offset:15849, original_length:16421, max_merged_trans_version:1726241621541056000, data_length:16421, data_zlength:16421, data_checksum:3599888316, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6604090, buf_size:16421, data_size:16421, row_count:158, column_count:8, max_merged_trans_version:1726241621541056000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1714289658, row_count_delta:2, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16421}) [2024-09-26 16:00:55.663580] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=18][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:55.663585] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=5][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:55.663588] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:55.663590] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=2][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:55.663593] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=3][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:55.663596] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=3][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:55.663599] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=3][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f92865a2118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92865a2118, limit_param:null, stmt_allocator:0x7f92865a27d8, allocator:0x7f92865a27d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337655631791, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:00:55.663716] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=117][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:55.663719] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=3][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f92865a2118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92865a2118, limit_param:null, stmt_allocator:0x7f92865a27d8, allocator:0x7f92865a27d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664ec360, local_datums:0x7f92664ec360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337655631791, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:00:55.663777] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=58][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:55.663780] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=2] partition merge iter row count(i=0, row_count=158, ghost_row_count=0, pkey={tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, table={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337655631791, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.663799] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=14][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:55.663812] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:55.663817] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=5][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:55.663820] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=3] task finish process(ret=-4184, start_time=1727337655662886, end_time=1727337655663820, runtime=934, *this={this:0x7f9264afa030, type:1, status:2, dag:{ObIDag:{this:0x7f927282cf60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D7-0-0, dag_ret:0, dag_status:2, start_time:1727337655652527, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:00:55.663830] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=10][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264afa030, type:1, status:2, dag:{ObIDag:{this:0x7f927282cf60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D7-0-0, dag_ret:0, dag_status:2, start_time:1727337655652527, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:55.663841] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=8] dag finished(dag_ret=-4184, runtime=11314, dag_cnt=37, dag_cnts_[dag.get_type()]=37, &dag=0x7f927282cf60, dag={ObIDag:{this:0x7f927282cf60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D7-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655652527, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}) [2024-09-26 16:00:55.663852] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D7-0-0] [lt=8] succeed to del sys task(removed_task={start_time:1727337655652509, task_id:YB42C0A8002D-0006230056D8C6D7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.663871] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337655663869, task_id:YB42C0A8002D-0006230056D8C6DC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.663875] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=4] schedule one task(task=0x7f927282ed10, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.663941] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1152921504606847408, range=0x7f92aa952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b0060, count:5, allocator:0x7f92b33d87d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684b0360, local_datums:0x7f92684b0360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:55.664291] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=18][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:55.664350] ERROR issue_dba_error (ob_log.cpp:1875) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=8][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.664355] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.664362] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:55.664365] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:55.664367] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 12, flag: 0, null: 0, ptr: 0x7f92684b4178, hex: 30323232362D32312D373825, cstr: 02226-21-78%},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f92684b4184, hex: 3230323430393133303435343039, cstr: 20240913045409},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b40e0, hex: CD7F500300000000, int: 55607245},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4118, hex: E8EC05CCA0600BE8, int: -1726179788432479000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-11869, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:238, row_store_type:0, opt:5, var_column_count:0, row_offset:15429, original_length:16321, max_merged_trans_version:1726190643638412000, data_length:16321, data_zlength:5388, data_checksum:152456761, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9262c6e030, buf_size:5388, data_size:16321, row_count:238, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2114797869, row_count_delta:238, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16321}) [2024-09-26 16:00:55.664406] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=38][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:55.664409] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=3][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:55.664412] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=3][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:55.664415] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=3][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:55.664418] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684b0360, local_datums:0x7f92684b0360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b03e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684b0360, local_datums:0x7f92684b0360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b03e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92b33d8118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b33d8118, limit_param:null, stmt_allocator:0x7f92b33d87d8, allocator:0x7f92b33d87d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684b0360, local_datums:0x7f92684b0360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b03e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337655631739, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:00:55.664506] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=88][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:55.664509] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=3][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684b0360, local_datums:0x7f92684b0360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b03e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92b33d8118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b33d8118, limit_param:null, stmt_allocator:0x7f92b33d87d8, allocator:0x7f92b33d87d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92684b0360, local_datums:0x7f92684b0360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b03e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337655631739, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:00:55.664559] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=49][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:55.664561] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=2] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337655631739, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.664577] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=13][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:55.664588] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=3][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:55.664593] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=5][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:55.664596] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=3] task finish process(ret=-4184, start_time=1727337655663896, end_time=1727337655664596, runtime=700, *this={this:0x7f9272926030, type:1, status:2, dag:{ObIDag:{this:0x7f927282bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D4-0-0, dag_ret:0, dag_status:2, start_time:1727337655642845, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.664614] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=17][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9272926030, type:1, status:2, dag:{ObIDag:{this:0x7f927282bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D4-0-0, dag_ret:0, dag_status:2, start_time:1727337655642845, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:00:55.664625] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=7] dag finished(dag_ret=-4184, runtime=21780, dag_cnt=36, dag_cnts_[dag.get_type()]=36, &dag=0x7f927282bf50, dag={ObIDag:{this:0x7f927282bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D4-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655642845, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.664636] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D4-0-0] [lt=8] succeed to del sys task(removed_task={start_time:1727337655642836, task_id:YB42C0A8002D-0006230056D8C6D4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.664655] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655664654, task_id:YB42C0A8002D-0006230056D8C6DD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.664659] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=4] schedule one task(task=0x7f927282f240, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.664687] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.664692] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.664696] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.664702] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.664707] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.664717] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.664745] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.664750] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.664753] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.664772] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.664777] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.664784] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.664819] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337655631836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.664833] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752918752, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828752918752, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.664851] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=14] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.664855] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.664863] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:309}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752918752, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.664890] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=21] succeed to build merge ctx(tablet_id={id:309}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752918752, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:309}, table_count:1, [{i:0, table_key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}, merge_scn:{val:1726241316899724000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8631da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655662195, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.67ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.664926] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=29] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282def0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DA-0-0, dag_ret:0, dag_status:2, start_time:1727337655661868, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}, scanned_row_cnt_arr:0x7f9263a45490, output_block_cnt_arr:0x7f9263a45498, concurrent_cnt:1, estimate_row_cnt:6, estimate_occupy_size:6291456, latest_update_ts:1727337655662195, estimated_finish_time:1727337672664929}) [2024-09-26 16:00:55.664935] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f927282e2b0, type:15, status:2, dag:{ObIDag:{this:0x7f927282def0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DA-0-0, dag_ret:0, dag_status:2, start_time:1727337655661868, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:55.664943] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=8] task finish process(ret=0, start_time=1727337655662191, end_time=1727337655664943, runtime=2752, *this={this:0x7f927282e2b0, type:15, status:2, dag:{ObIDag:{this:0x7f927282def0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DA-0-0, dag_ret:0, dag_status:2, start_time:1727337655661868, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:55.664981] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=2] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.665061] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=309, range=0x7f92aa6d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:11}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ee060, count:7, allocator:0x7f92a31fe7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92664ee3b0, local_datums:0x7f92664ee3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:55.665273] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337655631851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.665290] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.665315] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=19] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.665319] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.665326] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:331}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.665349] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:331}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:331}, table_count:1, [{i:0, table_key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}, merge_scn:{val:1726241465393532000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8633ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655662604, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.72ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.665380] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=20][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:55.665378] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282e420, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DB-0-0, dag_ret:0, dag_status:2, start_time:1727337655662477, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}, scanned_row_cnt_arr:0x7f9268515480, output_block_cnt_arr:0x7f9268515488, concurrent_cnt:1, estimate_row_cnt:45, estimate_occupy_size:0, latest_update_ts:1727337655662604, estimated_finish_time:1727337672665378}) [2024-09-26 16:00:55.665386] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f927282e7e0, type:15, status:2, dag:{ObIDag:{this:0x7f927282e420, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DB-0-0, dag_ret:0, dag_status:2, start_time:1727337655662477, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:55.665392] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=6] task finish process(ret=0, start_time=1727337655662602, end_time=1727337655665392, runtime=2790, *this={this:0x7f927282e7e0, type:15, status:2, dag:{ObIDag:{this:0x7f927282e420, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DB-0-0, dag_ret:0, dag_status:2, start_time:1727337655662477, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:55.665415] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=4] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.665445] ERROR issue_dba_error (ob_log.cpp:1875) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.665450] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.665459] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=9][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:00:55.665462] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:00:55.665465] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=3][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:7, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 12, flag: 0, null: 0, ptr: 0x7f9262c6e1e8, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e0e0, hex: 420B000000000000, int: 2882},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e118, hex: 0100000000000000, int: 1, num_digit0: 0},idx=4:{len: 16, flag: 0, null: 0, ptr: 0x7f9262c6e1f4, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e188, hex: C818154EB7520BE8, int: -1726195084923955000},idx=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e1c0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19292, column_count:11, rowkey_column_count:7, has_column_checksum:0, row_count:156, row_store_type:0, opt:0, var_column_count:0, row_offset:15827, original_length:16391, max_merged_trans_version:1726241316783508000, data_length:16391, data_zlength:16391, data_checksum:3629777942, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926fe04090, buf_size:16391, data_size:16391, row_count:156, column_count:11, max_merged_trans_version:1726241316783508000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:619385244, row_count_delta:1, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16391}) [2024-09-26 16:00:55.665482] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=16][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:00:55.665484] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:00:55.665474] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=331, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b56060, count:5, allocator:0x7f92bc8fa7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:55.665486] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:00:55.665489] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=3][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:00:55.665491] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:00:55.665493] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:00:55.665495] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92664ee3b0, local_datums:0x7f92664ee3b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee4d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee5b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee5f0, hex: 9C4DB16500000000, int: 1706118556},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:309}, iter_end:false, schema_rowkey_column_cnt:5, schema_version:1710828752918752, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92664ee3b0, local_datums:0x7f92664ee3b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee4d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee5b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee5f0, hex: 9C4DB16500000000, int: 1706118556},]}, store_ctx:{this:0x7f92a31fe118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31fe118, limit_param:null, stmt_allocator:0x7f92a31fe7d8, allocator:0x7f92a31fe7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241316899724000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92664ee3b0, local_datums:0x7f92664ee3b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee4d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee5b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee5f0, hex: 9C4DB16500000000, int: 1706118556},]}, key:{"BIGINT":1},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f9279a04108 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x7f927a839198 latest_compact_node=0x7f927aa30240 max_trans_version={val:1726241316783508000, v:0} max_trans_id=125581128 max_elr_trans_version={val:1726241316783508000, v:0} max_elr_trans_id=125581128 latest_compact_ts=1727334209251890 last_compact_cnt=189 total_trans_node_cnt=418 max_modify_scn={val:1726241316783508000, v:0} min_modify_scn={val:1726178872697529000, v:0}}, version_iter:this=0x7f927a8390e8 trans_version={val:1726241316783508000, v:0} scn={val:1726241316783508000, v:0} tx_id={txid:125581128} prev=0x7f927a838f78 next=0x7f927a839198 modify_count=196 acc_checksum=432508559 version=1726211894725631 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:65} seq_no=1726241316536989, multi_version_iter:this=0x7f927aa0b360 trans_version={val:1726194784763860000, v:0} scn={val:1726194784763860000, v:0} tx_id={txid:125216537} prev=0x7f927aa0b2b0 next=0x7f927aa0b9f8 modify_count=107 acc_checksum=2694663391 version=1726178872309143 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no=1726194784741342, max_committed_trans_version:1726241316783508000, cur_trans_version:{val:1726195084923955000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241316899724000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:156, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337655631836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:11, cur_pos:0, count:0}}) [2024-09-26 16:00:55.665601] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=106][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:00:55.665604] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=3][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:309}, iter_end:false, schema_rowkey_column_cnt:5, schema_version:1710828752918752, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92664ee3b0, local_datums:0x7f92664ee3b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee4d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee5b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee5f0, hex: 9C4DB16500000000, int: 1706118556},]}, store_ctx:{this:0x7f92a31fe118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31fe118, limit_param:null, stmt_allocator:0x7f92a31fe7d8, allocator:0x7f92a31fe7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241316899724000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92664ee3b0, local_datums:0x7f92664ee3b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee4d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee5b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee5f0, hex: 9C4DB16500000000, int: 1706118556},]}, key:{"BIGINT":1},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f9279a04108 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x7f927a839198 latest_compact_node=0x7f927aa30240 max_trans_version={val:1726241316783508000, v:0} max_trans_id=125581128 max_elr_trans_version={val:1726241316783508000, v:0} max_elr_trans_id=125581128 latest_compact_ts=1727334209251890 last_compact_cnt=189 total_trans_node_cnt=418 max_modify_scn={val:1726241316783508000, v:0} min_modify_scn={val:1726178872697529000, v:0}}, version_iter:this=0x7f927a8390e8 trans_version={val:1726241316783508000, v:0} scn={val:1726241316783508000, v:0} tx_id={txid:125581128} prev=0x7f927a838f78 next=0x7f927a839198 modify_count=196 acc_checksum=432508559 version=1726211894725631 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:65} seq_no=1726241316536989, multi_version_iter:this=0x7f927aa0b360 trans_version={val:1726194784763860000, v:0} scn={val:1726194784763860000, v:0} tx_id={txid:125216537} prev=0x7f927aa0b2b0 next=0x7f927aa0b9f8 modify_count=107 acc_checksum=2694663391 version=1726178872309143 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no=1726194784741342, max_committed_trans_version:1726241316783508000, cur_trans_version:{val:1726195084923955000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241316899724000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:156, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337655631836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:11, cur_pos:0, count:0}}]) [2024-09-26 16:00:55.665674] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=69][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:00:55.665677] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=2] partition merge iter row count(i=0, row_count=156, ghost_row_count=0, pkey={tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, table={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337655631836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.665693] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=12][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:00:55.665703] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:00:55.665703] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:00:55.665707] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:00:55.665710] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337655665020, end_time=1727337655665709, runtime=689, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f927282def0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DA-0-0, dag_ret:0, dag_status:2, start_time:1727337655661868, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:00:55.665719] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=8][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.665729] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=7] dag finished(dag_ret=-4184, runtime=3861, dag_cnt=35, dag_cnts_[dag.get_type()]=35, &dag=0x7f927282def0, dag={ObIDag:{this:0x7f927282def0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DA-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655661868, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}) [2024-09-26 16:00:55.665739] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DA-0-0] [lt=8] succeed to del sys task(removed_task={start_time:1727337655661857, task_id:YB42C0A8002D-0006230056D8C6DA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.665743] ERROR issue_dba_error (ob_log.cpp:1875) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.665746] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.665752] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=6][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.665765] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=1] partition merge iter row count(i=0, row_count=196, ghost_row_count=0, pkey={tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, table={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337655631851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.665777] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337655631758, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.665780] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337655631806, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.665788] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=10] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:00:55.665789] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655665428, end_time=1727337655665789, runtime=361, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f927282e420, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DB-0-0, dag_ret:0, dag_status:2, start_time:1727337655662477, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:00:55.665791] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92728f3480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.665792] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753247280, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828753247280, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.665800] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3322, dag_cnt=34, dag_cnts_[dag.get_type()]=34, &dag=0x7f927282e420, dag={ObIDag:{this:0x7f927282e420, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DB-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655662477, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}) [2024-09-26 16:00:55.665807] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DB-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655662454, task_id:YB42C0A8002D-0006230056D8C6DB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.665807] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.665817] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337655631865, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.665834] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337655665833, task_id:YB42C0A8002D-0006230056D8C6DE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.665838] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=4] schedule one task(task=0x7f927282f770, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.665844] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655665843, task_id:YB42C0A8002D-0006230056D8C6DF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.665846] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=2] schedule one task(task=0x7f927282fca0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.665838] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=20] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753357664, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753357664, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.665904] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3223, total_rows=22561) [2024-09-26 16:00:55.665908] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.665919] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=21] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:55.665914] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847409}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.665923] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.665931] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:1152921504606847409}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92728f3480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847409}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8640e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655651937, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=13.97ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.665931] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:458}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753357664, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.665959] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=21] succeed to build merge ctx(tablet_id={id:458}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753357664, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:458}, table_count:1, [{i:0, table_key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8627b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655664681, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.25ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.666042] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D5-0-0, dag_ret:0, dag_status:2, start_time:1727337655645320, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f35d0, output_block_cnt_arr:0x7f92728f35d8, concurrent_cnt:1, estimate_row_cnt:32439, estimate_occupy_size:18874368, latest_update_ts:1727337655651937, estimated_finish_time:1727337672666051}) [2024-09-26 16:00:55.666050] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f927282c840, type:15, status:2, dag:{ObIDag:{this:0x7f927282c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D5-0-0, dag_ret:0, dag_status:2, start_time:1727337655645320, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.666055] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=5] task finish process(ret=0, start_time=1727337655651934, end_time=1727337655666055, runtime=14121, *this={this:0x7f927282c840, type:15, status:2, dag:{ObIDag:{this:0x7f927282c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D5-0-0, dag_ret:0, dag_status:2, start_time:1727337655645320, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.666080] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=2] schedule one task(task=0x7f9272926030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.666124] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=29] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:00:55.666314] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282e950, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DC-0-0, dag_ret:0, dag_status:2, start_time:1727337655663875, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db54a0, output_block_cnt_arr:0x7f9287db54a8, concurrent_cnt:1, estimate_row_cnt:14598, estimate_occupy_size:4194304, latest_update_ts:1727337655664681, estimated_finish_time:1727337672666315}) [2024-09-26 16:00:55.666330] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=16] succeed to init merge ctx(task={this:0x7f927282ed10, type:15, status:2, dag:{ObIDag:{this:0x7f927282e950, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DC-0-0, dag_ret:0, dag_status:2, start_time:1727337655663875, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.666338] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=8] task finish process(ret=0, start_time=1727337655664678, end_time=1727337655666338, runtime=1660, *this={this:0x7f927282ed10, type:15, status:2, dag:{ObIDag:{this:0x7f927282e950, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DC-0-0, dag_ret:0, dag_status:2, start_time:1727337655663875, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.666359] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=3] schedule one task(task=0x7f9272838f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.666376] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.666380] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.666383] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.666389] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.666393] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.666402] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.666401] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=458, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b0060, count:8, allocator:0x7f928642a7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92684b03e0, local_datums:0x7f92684b03e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:00:55.666428] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.666432] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.666434] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.666438] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.666441] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.666447] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.666523] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1152921504606847409, range=0x7f92aaf52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665ac060, count:5, allocator:0x7f9269ffc7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92665ac360, local_datums:0x7f92665ac360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:00:55.666646] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.666716] ERROR issue_dba_error (ob_log.cpp:1875) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.666719] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.666729] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=0] partition merge iter row count(i=0, row_count=247, ghost_row_count=0, pkey={tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337655631865, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.666749] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655666371, end_time=1727337655666749, runtime=378, *this={this:0x7f9272838f50, type:1, status:2, dag:{ObIDag:{this:0x7f927282e950, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DC-0-0, dag_ret:0, dag_status:2, start_time:1727337655663875, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.666758] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2883, dag_cnt=33, dag_cnts_[dag.get_type()]=33, &dag=0x7f927282e950, dag={ObIDag:{this:0x7f927282e950, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DC-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655663875, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.666765] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DC-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655663869, task_id:YB42C0A8002D-0006230056D8C6DC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.666798] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337655666797, task_id:YB42C0A8002D-0006230056D8C6E0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.666801] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=3] schedule one task(task=0x7f92728301d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.666904] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.666955] ERROR issue_dba_error (ob_log.cpp:1875) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.666959] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.666971] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=0] partition merge iter row count(i=0, row_count=250, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337655631758, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.666998] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655666472, end_time=1727337655666998, runtime=526, *this={this:0x7f9272926030, type:1, status:2, dag:{ObIDag:{this:0x7f927282c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D5-0-0, dag_ret:0, dag_status:2, start_time:1727337655645320, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.667011] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=21690, dag_cnt=32, dag_cnts_[dag.get_type()]=32, &dag=0x7f927282c480, dag={ObIDag:{this:0x7f927282c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D5-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655645320, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.667022] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D5-0-0] [lt=8] succeed to del sys task(removed_task={start_time:1727337655645303, task_id:YB42C0A8002D-0006230056D8C6D5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.667039] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655667038, task_id:YB42C0A8002D-0006230056D8C6E1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.667042] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=2] schedule one task(task=0x7f9272830700, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.667078] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.667083] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.667086] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.667091] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.667096] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=4] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b862a0a0, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:490, tablet_addr:[816017-1849-0-2305843629838340096-2](offset=1871872,size=578,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:472}, data_tablet_id:{id:472}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:19976489, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862a734, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844179594129408-0](offset=1847296,size=1090,type=2,seq=0), type:"MAJOR", tablet_id:{id:472}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389523010000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843910084932674-0](offset=1848386,size=839,type=2,seq=0), type:"MINOR", tablet_id:{id:472}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178194887823000, upper_trans_version:1726178194887823000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337655631939, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd4080, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209505593}, this:0x7f92a3bd4080, timestamp:1727334209505593, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:65, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274254072414000, v:0}, rec_scn:{val:1726241829658678000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697839104-2](offset=1859584,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834376524-0](offset=1859916,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881206236-0](offset=1860060,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447726270-0](offset=1860286,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7ac60, 0x7f92a3bd4080, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.667158] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.667166] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.667194] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.667198] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.667200] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.667204] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.667208] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b863fe60, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:511, tablet_addr:[816017-1849-0-2305843652386852864-2](offset=1806336,size=599,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:501}, data_tablet_id:{id:501}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1792252290, row_count:320564}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8640560, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305866008765333504-0](offset=1769472,size=21420,type=2,seq=0), type:"MAJOR", tablet_id:{id:501}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389518729000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844002426672044-0](offset=1790892,size=925,type=2,seq=0), type:"MINOR", tablet_id:{id:501}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178194873791000, upper_trans_version:1726178194873791000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337655631924, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd03d0, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495744}, this:0x7f92a3bd03d0, timestamp:1727334209495744, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:58, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274254042757000, v:0}, rec_scn:{val:1726241824983317000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771515392-2](offset=1794048,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161663821-0](offset=1794381,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881140719-0](offset=1794543,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843215373918929-0](offset=1794769,size=192,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a78d20, 0x7f92a3bd03d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.667254] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=46] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.667260] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.667274] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337655631909, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.667285] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753437040, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=57, max_schema_version_in_memtable=1710828753437040, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.667404] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=20] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1362, total_rows=1362) [2024-09-26 16:00:55.667410] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.667419] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:492}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:12, rowkey_column_count:11, schema_rowkey_col_cnt:9, full_stored_col_cnt:59, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753437040, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=28 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.667480] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=54] succeed to build merge ctx(tablet_id={id:492}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753437040, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:492}, table_count:1, [{i:0, table_key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8624c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655666424, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=992us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.667605] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=29] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282f8e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DF-0-0, dag_ret:0, dag_status:2, start_time:1727337655665846, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a454c0, output_block_cnt_arr:0x7f9263a454c8, concurrent_cnt:1, estimate_row_cnt:1362, estimate_occupy_size:2097152, latest_update_ts:1727337655666424, estimated_finish_time:1727337672667606}) [2024-09-26 16:00:55.667619] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f927282fca0, type:15, status:2, dag:{ObIDag:{this:0x7f927282f8e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DF-0-0, dag_ret:0, dag_status:2, start_time:1727337655665846, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.667627] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=7] task finish process(ret=0, start_time=1727337655666423, end_time=1727337655667627, runtime=1204, *this={this:0x7f927282fca0, type:15, status:2, dag:{ObIDag:{this:0x7f927282f8e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DF-0-0, dag_ret:0, dag_status:2, start_time:1727337655665846, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.667667] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=1] schedule one task(task=0x7f9272838f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.667714] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337655631894, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.667728] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753332736, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=36, max_schema_version_in_memtable=1710828753332736, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.667737] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=492, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:57, schema_rowkey_cnt:9, rowkey_cnt:11, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:9, column_cnt:59}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:11, allocator:0x7f92636207d8, capacity:11, init_cnt:11}, datum_utils:{is_oracle_mode:false, rowkey_cnt:11, col_cnt:11, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:59, datum_buffer:{capacity:59, datums:0x7f92665fe050, local_datums:0x7f92684b4440}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,col_id=43:NOP,col_id=44:NOP,col_id=45:NOP,col_id=46:NOP,col_id=47:NOP,col_id=48:NOP,col_id=49:NOP,col_id=50:NOP,col_id=51:NOP,col_id=52:NOP,col_id=53:NOP,col_id=54:NOP,col_id=55:NOP,col_id=56:NOP,col_id=57:NOP,col_id=58:NOP,]}) [2024-09-26 16:00:55.667921] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=12] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:55.667926] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.667950] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=18][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.667933] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:455}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:10, rowkey_column_count:9, schema_rowkey_col_cnt:7, full_stored_col_cnt:38, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753332736, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=26 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.667986] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.667967] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=25] succeed to build merge ctx(tablet_id={id:455}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753332736, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:455}, table_count:1, [{i:0, table_key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655666369, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.56ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.667989] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.667998] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=0] partition merge iter row count(i=0, row_count=61, ghost_row_count=0, pkey={tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337655631909, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.668019] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655667680, end_time=1727337655668019, runtime=339, *this={this:0x7f9272838f50, type:1, status:2, dag:{ObIDag:{this:0x7f927282f8e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DF-0-0, dag_ret:0, dag_status:2, start_time:1727337655665846, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.668029] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2183, dag_cnt=31, dag_cnts_[dag.get_type()]=31, &dag=0x7f927282f8e0, dag={ObIDag:{this:0x7f927282f8e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DF-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655665846, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.668036] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DF-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655665843, task_id:YB42C0A8002D-0006230056D8C6DF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.668057] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337655668055, task_id:YB42C0A8002D-0006230056D8C6E2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.668060] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=2] schedule one task(task=0x7f9272830c30, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.668077] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.668080] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.668082] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726077706339704027) [2024-09-26 16:00:55.668089] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.668093] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726077706339704027, tablet={this:0x7f92b864a0a0, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[720905-1289-0-2305843650239553536-1](offset=1990656,size=597,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:342}, data_tablet_id:{id:342}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725843677180998000, v:0}, clog_checkpoint_scn:{val:1725843677180998000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726077706339704027, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2338495768, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864a7a0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[720905-1289-0-2305844556477644800-0](offset=1982464,size=1441,type=2,seq=0), type:"MAJOR", tablet_id:{id:342}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725843393342764001}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337655631954, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[720905-1289-0-2305843251881332736-1](offset=1986560,size=226,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[720905-1289-0-2305843183161856226-0](offset=1986786,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[720905-1289-0-2305843251881333124-0](offset=1986948,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[720905-1289-0-2305843216447853158-0](offset=1987174,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7cba0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:55.668127] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=34] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.668134] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.668191] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=25] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:00:55.668415] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282f3b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DE-0-0, dag_ret:0, dag_status:2, start_time:1727337655665838, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d4b0, output_block_cnt_arr:0x7f9264b1d4b8, concurrent_cnt:1, estimate_row_cnt:3824, estimate_occupy_size:4194304, latest_update_ts:1727337655666369, estimated_finish_time:1727337672668417}) [2024-09-26 16:00:55.668430] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=15] succeed to init merge ctx(task={this:0x7f927282f770, type:15, status:2, dag:{ObIDag:{this:0x7f927282f3b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DE-0-0, dag_ret:0, dag_status:2, start_time:1727337655665838, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.668438] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=8] task finish process(ret=0, start_time=1727337655666367, end_time=1727337655668438, runtime=2071, *this={this:0x7f927282f770, type:15, status:2, dag:{ObIDag:{this:0x7f927282f3b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DE-0-0, dag_ret:0, dag_status:2, start_time:1727337655665838, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.668515] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=2] schedule one task(task=0x7f92728392e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.668560] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:55.668629] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=455, range=0x7f92aaed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:36, schema_rowkey_cnt:7, rowkey_cnt:9, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:7, column_cnt:38}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a60060, count:9, allocator:0x7f9275d767d8, capacity:9, init_cnt:9}, datum_utils:{is_oracle_mode:false, rowkey_cnt:9, col_cnt:9, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:38, datum_buffer:{capacity:38, datums:0x7f9264a61340, local_datums:0x7f9264a603f0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,]}) [2024-09-26 16:00:55.668880] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=18][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.668943] ERROR issue_dba_error (ob_log.cpp:1875) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.668948] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.668959] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=0] partition merge iter row count(i=0, row_count=140, ghost_row_count=0, pkey={tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337655631894, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.668984] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655668584, end_time=1727337655668984, runtime=400, *this={this:0x7f92728392e0, type:1, status:2, dag:{ObIDag:{this:0x7f927282f3b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DE-0-0, dag_ret:0, dag_status:2, start_time:1727337655665838, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.668994] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3156, dag_cnt=30, dag_cnts_[dag.get_type()]=30, &dag=0x7f927282f3b0, dag={ObIDag:{this:0x7f927282f3b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DE-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655665838, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.668994] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:55.669001] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337655668992) [2024-09-26 16:00:55.669002] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DE-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655665833, task_id:YB42C0A8002D-0006230056D8C6DE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.669004] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337655568979, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:55.669024] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223384898196, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.669029] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:55.669035] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:55.669039] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:55.669042] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:55.669041] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337655631880, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.669048] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.669050] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.669052] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337655669007) [2024-09-26 16:00:55.669050] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753417144, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828753417144, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.669058] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337655669057, task_id:YB42C0A8002D-0006230056D8C6E3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.669062] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=4] schedule one task(task=0x7f9272831160, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.669139] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:55.669144] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.669150] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:487}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753417144, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.669171] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:487}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753417144, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:487}, table_count:1, [{i:0, table_key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655664740, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.41ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.669260] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=19] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:00:55.669351] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=2] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282ee80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DD-0-0, dag_ret:0, dag_status:2, start_time:1727337655664659, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab4a0, output_block_cnt_arr:0x7f92684ab4a8, concurrent_cnt:1, estimate_row_cnt:2199, estimate_occupy_size:0, latest_update_ts:1727337655664740, estimated_finish_time:1727337672669350}) [2024-09-26 16:00:55.669358] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f927282f240, type:15, status:2, dag:{ObIDag:{this:0x7f927282ee80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DD-0-0, dag_ret:0, dag_status:2, start_time:1727337655664659, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.669364] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=5] task finish process(ret=0, start_time=1727337655664739, end_time=1727337655669364, runtime=4625, *this={this:0x7f927282f240, type:15, status:2, dag:{ObIDag:{this:0x7f927282ee80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DD-0-0, dag_ret:0, dag_status:2, start_time:1727337655664659, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.669392] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=4] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.669409] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.669413] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.669415] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.669419] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.669422] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.669440] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.669434] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=487, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:11}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:8, allocator:0x7f92c183a7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92664f03e0, local_datums:0x7f92664f03e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:00:55.669457] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337655631924, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.669467] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753441792, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828753441792, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.669540] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1295, total_rows=1295) [2024-09-26 16:00:55.669544] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.669549] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:501}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753441792, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.669566] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:501}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753441792, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:501}, table_count:1, [{i:0, table_key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655667189, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.36ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.669645] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282fe10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E0-0-0, dag_ret:0, dag_status:2, start_time:1727337655666801, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b21490, output_block_cnt_arr:0x7f9264b21498, concurrent_cnt:1, estimate_row_cnt:1295, estimate_occupy_size:0, latest_update_ts:1727337655667189, estimated_finish_time:1727337672669644}) [2024-09-26 16:00:55.669652] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f92728301d0, type:15, status:2, dag:{ObIDag:{this:0x7f927282fe10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E0-0-0, dag_ret:0, dag_status:2, start_time:1727337655666801, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.669657] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=5] task finish process(ret=0, start_time=1727337655667187, end_time=1727337655669657, runtime=2470, *this={this:0x7f92728301d0, type:15, status:2, dag:{ObIDag:{this:0x7f927282fe10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E0-0-0, dag_ret:0, dag_status:2, start_time:1727337655666801, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.669684] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337655631968, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.669693] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.669701] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=5] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.669703] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.669710] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.669707] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:140}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.669722] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:140}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:140}, table_count:1, [{i:0, table_key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8622d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655669403, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=303us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.669740] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272830da0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E3-0-0, dag_ret:0, dag_status:2, start_time:1727337655669062, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b23460, output_block_cnt_arr:0x7f9264b23468, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337655669403, estimated_finish_time:1727337672669739}) [2024-09-26 16:00:55.669745] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=4] succeed to init merge ctx(task={this:0x7f9272831160, type:15, status:2, dag:{ObIDag:{this:0x7f9272830da0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E3-0-0, dag_ret:0, dag_status:2, start_time:1727337655669062, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.669749] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=4] task finish process(ret=0, start_time=1727337655669401, end_time=1727337655669749, runtime=348, *this={this:0x7f9272831160, type:15, status:2, dag:{ObIDag:{this:0x7f9272830da0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E3-0-0, dag_ret:0, dag_status:2, start_time:1727337655669062, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.669758] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.669760] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.669761] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.669764] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:412}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753247280, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.669772] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=0] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337655631880, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.669778] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:412}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753247280, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:412}, table_count:1, [{i:0, table_key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655654593, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=15.17ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.669793] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=12] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f927282d490, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D8-0-0, dag_ret:0, dag_status:2, start_time:1727337655654449, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1470, output_block_cnt_arr:0x7f92728f1478, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337655654593, estimated_finish_time:1727337672669796}) [2024-09-26 16:00:55.669792] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655669405, end_time=1727337655669792, runtime=387, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f927282ee80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DD-0-0, dag_ret:0, dag_status:2, start_time:1727337655664659, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.669798] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f927282d850, type:15, status:2, dag:{ObIDag:{this:0x7f927282d490, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D8-0-0, dag_ret:0, dag_status:2, start_time:1727337655654449, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.669802] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=4] task finish process(ret=0, start_time=1727337655654591, end_time=1727337655669802, runtime=15211, *this={this:0x7f927282d850, type:15, status:2, dag:{ObIDag:{this:0x7f927282d490, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D8-0-0, dag_ret:0, dag_status:2, start_time:1727337655654449, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.669802] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5143, dag_cnt=29, dag_cnts_[dag.get_type()]=29, &dag=0x7f927282ee80, dag={ObIDag:{this:0x7f927282ee80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6DD-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655664659, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.669809] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6DD-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655664654, task_id:YB42C0A8002D-0006230056D8C6DD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.669812] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337655631939, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.669819] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753360568, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753360568, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.669834] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=4] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.669836] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=14] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.669839] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=3] schedule one task(task=0x7f9264b46030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.669844] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337655669843, task_id:YB42C0A8002D-0006230056D8C6E4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.669847] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=2] schedule one task(task=0x7f9272831690, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.669889] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=501, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265a7c060, count:6, allocator:0x7f92865507d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265a7c390, local_datums:0x7f9265a7c390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:55.669903] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=140, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265a80060, count:3, allocator:0x7f92636947d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9265a80320, local_datums:0x7f9265a80320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:55.669955] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.669960] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.669990] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.669992] ERROR issue_dba_error (ob_log.cpp:1875) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.669992] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.669995] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.670000] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=0] partition merge iter row count(i=0, row_count=56, ghost_row_count=0, pkey={tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337655631924, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.670005] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337655631968, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.670016] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655669859, end_time=1727337655670016, runtime=157, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f927282fe10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E0-0-0, dag_ret:0, dag_status:2, start_time:1727337655666801, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.670025] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3224, dag_cnt=28, dag_cnts_[dag.get_type()]=28, &dag=0x7f927282fe10, dag={ObIDag:{this:0x7f927282fe10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E0-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655666801, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.670025] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655669865, end_time=1727337655670025, runtime=160, *this={this:0x7f9264b46030, type:1, status:2, dag:{ObIDag:{this:0x7f9272830da0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E3-0-0, dag_ret:0, dag_status:2, start_time:1727337655669062, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.670032] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E0-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655666797, task_id:YB42C0A8002D-0006230056D8C6E0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.670051] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337655670050, task_id:YB42C0A8002D-0006230056D8C6E5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.670054] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=2] schedule one task(task=0x7f9272831bc0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.670055] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.670059] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.670061] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.670065] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.670068] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8641da0, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:487, tablet_addr:[816017-1849-0-2305843626617249792-2](offset=2007040,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101009}, data_tablet_id:{id:140}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726073917016589000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3251381279, row_count:150}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8642458, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844188184215552-0](offset=1998848,size=1098,type=2,seq=0), type:"MAJOR", tablet_id:{id:101009}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726070403667750001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843957329724490-0](offset=1999946,size=883,type=2,seq=0), type:"MINOR", tablet_id:{id:101009}, scn_range:{start_scn:{val:1726073917016589000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726167395874925000, upper_trans_version:1726167395874925000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337655631982, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd0ba0, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495816}, this:0x7f92a3bd0ba0, timestamp:1727334209495816, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:59, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726256203821911000, v:0}, rec_scn:{val:1726241803398655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697982464-2](offset=2002944,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276970828-0](offset=2003276,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881349602-0](offset=2003426,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7bc00, 0x7f92a3bd0ba0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.670107] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=39] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.670114] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.670112] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.670118] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.670121] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:00:55.670130] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=2] schedule one task(task=0x7f9272838f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.670137] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=12] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.670160] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.670168] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:472}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753360568, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.670142] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8642d40, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:487, tablet_addr:[816017-1849-0-2305843626617085952-2](offset=1843200,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101010}, data_tablet_id:{id:140}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726073917016589000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4294280206, row_count:150}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86433f8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844211806371840-0](offset=1835008,size=1120,type=2,seq=0), type:"MAJOR", tablet_id:{id:101010}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726070403667750001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843968066978912-0](offset=1836128,size=893,type=2,seq=0), type:"MINOR", tablet_id:{id:101010}, scn_range:{start_scn:{val:1726073917016589000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726167395874925000, upper_trans_version:1726167395874925000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337655631996, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd1370, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495895}, this:0x7f92a3bd1370, timestamp:1727334209495895, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:60, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726256203821911000, v:0}, rec_scn:{val:1726241803398655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697818624-2](offset=1839104,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276806988-0](offset=1839436,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881185762-0](offset=1839586,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7c3d0, 0x7f92a3bd1370, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.670192] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=50] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.670185] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:472}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753360568, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:472}, table_count:1, [{i:0, table_key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655667072, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.09ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.670200] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.670205] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272830340, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E1-0-0, dag_ret:0, dag_status:2, start_time:1727337655667041, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f460, output_block_cnt_arr:0x7f9264b1f468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337655667072, estimated_finish_time:1727337672670204}) [2024-09-26 16:00:55.670211] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272830700, type:15, status:2, dag:{ObIDag:{this:0x7f9272830340, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E1-0-0, dag_ret:0, dag_status:2, start_time:1727337655667041, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.670216] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=4] task finish process(ret=0, start_time=1727337655667070, end_time=1727337655670216, runtime=3146, *this={this:0x7f9272830700, type:15, status:2, dag:{ObIDag:{this:0x7f9272830340, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E1-0-0, dag_ret:0, dag_status:2, start_time:1727337655667041, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.670233] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=12] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.670247] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1185, dag_cnt=27, dag_cnts_[dag.get_type()]=27, &dag=0x7f9272830da0, dag={ObIDag:{this:0x7f9272830da0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E3-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655669062, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.670256] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E3-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655669057, task_id:YB42C0A8002D-0006230056D8C6E3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.670270] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337655670268, task_id:YB42C0A8002D-0006230056D8C6E6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.670273] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=3] schedule one task(task=0x7f92728320f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.670271] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=412, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:7}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a36060, count:4, allocator:0x7f92636207d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9263a36350, local_datums:0x7f9263a36350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:55.670306] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=22][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.670362] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.670366] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.670375] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=0] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337655631806, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.670411] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655670216, end_time=1727337655670411, runtime=195, *this={this:0x7f9272838f50, type:1, status:2, dag:{ObIDag:{this:0x7f927282d490, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D8-0-0, dag_ret:0, dag_status:2, start_time:1727337655654449, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.670422] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=15973, dag_cnt=26, dag_cnts_[dag.get_type()]=26, &dag=0x7f927282d490, dag={ObIDag:{this:0x7f927282d490, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6D8-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655654449, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.670431] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6D8-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655654444, task_id:YB42C0A8002D-0006230056D8C6D8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.670495] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655670494, task_id:YB42C0A8002D-0006230056D8C6E7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.670499] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=3] schedule one task(task=0x7f9272832620, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.670514] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337655631996, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.670525] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.670538] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.670542] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.670547] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101010}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:55.670563] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101010}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101010}, table_count:1, [{i:0, table_key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8642d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655670105, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=440us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.670586] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272831800, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E5-0-0, dag_ret:0, dag_status:2, start_time:1727337655670054, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5470, output_block_cnt_arr:0x7f9287db5478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337655670105, estimated_finish_time:1727337672670586}) [2024-09-26 16:00:55.670593] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9272831bc0, type:15, status:2, dag:{ObIDag:{this:0x7f9272831800, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E5-0-0, dag_ret:0, dag_status:2, start_time:1727337655670054, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.670599] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=5] task finish process(ret=0, start_time=1727337655670103, end_time=1727337655670599, runtime=496, *this={this:0x7f9272831bc0, type:15, status:2, dag:{ObIDag:{this:0x7f9272831800, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E5-0-0, dag_ret:0, dag_status:2, start_time:1727337655670054, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.670632] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=2] schedule one task(task=0x7f9264b46030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.670643] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.670647] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.670649] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:55.670653] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.670677] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101010, range=0x7f92a98d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:4, allocator:0x7f9275d787d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a62350, local_datums:0x7f9264a62350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:55.670657] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726112893187579000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864b040, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[812168-2208-0-2305843644869742592-2](offset=888832,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:323}, data_tablet_id:{id:323}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726112893187579000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2130074741, row_count:1002}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864b71c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[812168-2208-0-2305844507084419072-0](offset=880640,size=1395,type=2,seq=0), type:"MAJOR", tablet_id:{id:323}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562725180000}}, minor_tables:{{i:0, addr:[812168-2208-0-2305843933706286451-0](offset=882035,size=861,type=2,seq=0), type:"MINI", tablet_id:{id:323}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726112893187579000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726112601715820000, upper_trans_version:1726112601715820000}, {i:1, addr:[812168-2208-0-2305843933706287312-0](offset=882896,size=861,type=2,seq=0), type:"MINI", tablet_id:{id:323}, scn_range:{start_scn:{val:1726112893187579000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149985954467000, upper_trans_version:1726149985954467000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337655632011, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd9e40, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220581091}, this:0x7f92a3bd9e40, timestamp:1727334220581091, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:77, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265243868557000, v:0}, rec_scn:{val:1726264808086409001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[812168-2208-0-2305843482734723072-2](offset=884736,size=441,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[812168-2208-0-2305843176718303673-0](offset=885177,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843251880231509-0](offset=885333,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843216446751543-0](offset=885559,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7d370, 0x7f92a3bd9e40, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.670696] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=38] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.670702] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.670710] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.670736] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.670739] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.670740] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:55.670742] ERROR issue_dba_error (ob_log.cpp:1875) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.670743] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.670745] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.670744] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.670749] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.670753] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337655631996, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.670770] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655670644, end_time=1727337655670770, runtime=126, *this={this:0x7f9264b46030, type:1, status:2, dag:{ObIDag:{this:0x7f9272831800, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E5-0-0, dag_ret:0, dag_status:2, start_time:1727337655670054, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.670779] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=725, dag_cnt=25, dag_cnts_[dag.get_type()]=25, &dag=0x7f9272831800, dag={ObIDag:{this:0x7f9272831800, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E5-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655670054, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.670785] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E5-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655670050, task_id:YB42C0A8002D-0006230056D8C6E5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.670802] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337655670801, task_id:YB42C0A8002D-0006230056D8C6E8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.670792] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=8] multi version scan iterator init succ(param.table_id_=472, range=0x7f92aab52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287db6060, count:3, allocator:0x7f92b33fe7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f9287db6320, local_datums:0x7f9287db6320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:55.670805] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=2] schedule one task(task=0x7f9272832b50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.670840] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.670895] ERROR issue_dba_error (ob_log.cpp:1875) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.670899] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.670908] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=1] partition merge iter row count(i=0, row_count=17, ghost_row_count=0, pkey={tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337655631939, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.670929] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655670762, end_time=1727337655670928, runtime=166, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9272830340, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E1-0-0, dag_ret:0, dag_status:2, start_time:1727337655667041, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.670938] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3897, dag_cnt=24, dag_cnts_[dag.get_type()]=24, &dag=0x7f9272830340, dag={ObIDag:{this:0x7f9272830340, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E1-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655667041, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.670946] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E1-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655667038, task_id:YB42C0A8002D-0006230056D8C6E1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.670960] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655670959, task_id:YB42C0A8002D-0006230056D8C6E9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.670964] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=3] schedule one task(task=0x7f9272833080, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.670974] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.670977] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.670979] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:55.670981] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.670983] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.670985] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.670985] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.670987] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:55.670991] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.670991] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.670994] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706339704000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864bfe0, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:499, tablet_addr:[809012-1445-0-2305843639500640256-2](offset=495616,size=587,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101065}, data_tablet_id:{id:332}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706339704000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2355617684, row_count:171}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864c6e0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844381456211968-0](offset=466944,size=1278,type=2,seq=0), type:"MAJOR", tablet_id:{id:101065}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843926189679870-0](offset=468222,size=854,type=2,seq=0), type:"MINI", tablet_id:{id:101065}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337655632054, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdc550, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220895690}, this:0x7f92a3bdc550, timestamp:1727334220895690, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:82, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265274283037001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696450560-2](offset=471040,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843183160340812-0](offset=471372,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879817710-0](offset=471534,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7db40, 0x7f92a3bdc550, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.671033] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=38] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.671039] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.671228] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337655632040, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.671245] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=17] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=16, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.671263] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=12] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.671267] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.671273] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:297}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:18, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.671305] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=25] succeed to build merge ctx(tablet_id={id:297}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:297}, table_count:1, [{i:0, table_key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655670970, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=300us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.671345] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=34] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272832790, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E8-0-0, dag_ret:0, dag_status:2, start_time:1727337655670805, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab490, output_block_cnt_arr:0x7f92684ab498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337655670970, estimated_finish_time:1727337672671345}) [2024-09-26 16:00:55.671353] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9272832b50, type:15, status:2, dag:{ObIDag:{this:0x7f9272832790, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E8-0-0, dag_ret:0, dag_status:2, start_time:1727337655670805, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.671374] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=20] task finish process(ret=0, start_time=1727337655670969, end_time=1727337655671373, runtime=404, *this={this:0x7f9272832b50, type:15, status:2, dag:{ObIDag:{this:0x7f9272832790, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E8-0-0, dag_ret:0, dag_status:2, start_time:1727337655670805, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.671389] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=2] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.671405] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337655632054, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.671414] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.671427] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.671431] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.671446] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101065}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.671471] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=11] multi version scan iterator init succ(param.table_id_=297, range=0x7f92aaad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:16, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:18}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:6, allocator:0x7f92a31f67d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:18, datum_buffer:{capacity:32, datums:0x7f9263a3a390, local_datums:0x7f9263a3a390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,]}) [2024-09-26 16:00:55.671478] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=26] succeed to build merge ctx(tablet_id={id:101065}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101065}, table_count:1, [{i:0, table_key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655670976, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=458us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.671501] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.671503] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272832cc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E9-0-0, dag_ret:0, dag_status:2, start_time:1727337655670963, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad490, output_block_cnt_arr:0x7f92684ad498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337655670976, estimated_finish_time:1727337672671502}) [2024-09-26 16:00:55.671509] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272833080, type:15, status:2, dag:{ObIDag:{this:0x7f9272832cc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E9-0-0, dag_ret:0, dag_status:2, start_time:1727337655670963, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.671514] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=5] task finish process(ret=0, start_time=1727337655670974, end_time=1727337655671514, runtime=540, *this={this:0x7f9272833080, type:15, status:2, dag:{ObIDag:{this:0x7f9272832cc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E9-0-0, dag_ret:0, dag_status:2, start_time:1727337655670963, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.671527] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=3] schedule one task(task=0x7f9268486030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.671551] ERROR issue_dba_error (ob_log.cpp:1875) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.671555] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.671565] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337655632040, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.671587] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655671411, end_time=1727337655671587, runtime=176, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9272832790, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E8-0-0, dag_ret:0, dag_status:2, start_time:1727337655670805, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.671597] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=792, dag_cnt=23, dag_cnts_[dag.get_type()]=23, &dag=0x7f9272832790, dag={ObIDag:{this:0x7f9272832790, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E8-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655670805, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.671605] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E8-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655670801, task_id:YB42C0A8002D-0006230056D8C6E8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.671620] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337655671618, task_id:YB42C0A8002D-0006230056D8C6EA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.671623] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=3] schedule one task(task=0x7f92728335b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.671680] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101065, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:6, allocator:0x7f92864e07d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9263a3a390, local_datums:0x7f9263a3a390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:55.671711] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.671741] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.671744] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.671753] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=0] partition merge iter row count(i=0, row_count=15, ghost_row_count=0, pkey={tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337655632054, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.671784] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655671653, end_time=1727337655671784, runtime=131, *this={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9272832cc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E9-0-0, dag_ret:0, dag_status:2, start_time:1727337655670963, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.671795] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=832, dag_cnt=22, dag_cnts_[dag.get_type()]=22, &dag=0x7f9272832cc0, dag={ObIDag:{this:0x7f9272832cc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E9-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655670963, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.671802] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E9-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655670959, task_id:YB42C0A8002D-0006230056D8C6E9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.671814] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655671813, task_id:YB42C0A8002D-0006230056D8C6EB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.671817] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=3] schedule one task(task=0x7f9272833ae0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.671846] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.671850] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.671855] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=5][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:55.671860] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.671863] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077692196634026, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8653ce0, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:505, tablet_addr:[809012-1445-0-2305843645943099392-2](offset=503808,size=593,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101066}, data_tablet_id:{id:333}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077692196634026, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3243687120, row_count:2118}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8654404, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844488830410752-0](offset=483328,size=1378,type=2,seq=0), type:"MAJOR", tablet_id:{id:101066}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843931558405474-0](offset=484706,size=859,type=2,seq=0), type:"MINI", tablet_id:{id:101066}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337655632084, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdd4f0, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221023266}, this:0x7f92a3bdd4f0, timestamp:1727334221023266, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:84, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265274283037001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696471040-2](offset=491520,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843189602812236-0](offset=491852,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879838196-0](offset=492020,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5020, 0x7f92a3bdd4f0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.671910] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=47] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.671917] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.671964] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.671967] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.671969] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:55.671972] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.671975] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706139416000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8651da0, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:516, tablet_addr:[809012-1445-0-2305843657754255360-2](offset=499712,size=604,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:333}, data_tablet_id:{id:333}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706139416000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2218031597, row_count:2118}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86524c4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844845312692224-0](offset=479232,size=1710,type=2,seq=0), type:"MAJOR", tablet_id:{id:333}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843972360591022-0](offset=480942,size=897,type=2,seq=0), type:"MINI", tablet_id:{id:333}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337655632069, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdcd20, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220937046}, this:0x7f92a3bdcd20, timestamp:1727334220937046, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:83, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265274283037001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843367843950592-2](offset=487424,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843189602808142-0](offset=487758,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879834102-0](offset=487926,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446354136-0](offset=488152,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc4850, 0x7f92a3bdcd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.672007] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=32] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.672011] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.672158] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337655632084, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.672170] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.672184] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.672187] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.672193] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101066}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.672212] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101066}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101066}, table_count:1, [{i:0, table_key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8653ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655671841, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=350us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.672256] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272833720, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EB-0-0, dag_ret:0, dag_status:2, start_time:1727337655671817, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a43490, output_block_cnt_arr:0x7f9263a43498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337655671841, estimated_finish_time:1727337672672256}) [2024-09-26 16:00:55.672264] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9272833ae0, type:15, status:2, dag:{ObIDag:{this:0x7f9272833720, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EB-0-0, dag_ret:0, dag_status:2, start_time:1727337655671817, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.672269] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=5] task finish process(ret=0, start_time=1727337655671839, end_time=1727337655672269, runtime=430, *this={this:0x7f9272833ae0, type:15, status:2, dag:{ObIDag:{this:0x7f9272833720, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EB-0-0, dag_ret:0, dag_status:2, start_time:1727337655671817, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.672291] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=2] schedule one task(task=0x7f9272926030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.673692] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337655631954, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.673712] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=20] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753001832, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=21, max_schema_version_in_memtable=1710828753001832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.673687] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337655632069, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.673723] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.673726] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.673722] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=34] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.673731] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:342}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:23, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753001832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.673746] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=12] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.673750] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.673748] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337655631982, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.673749] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:342}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753001832, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:342}, table_count:1, [{i:0, table_key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655668072, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.66ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.673764] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.673771] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272830870, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E2-0-0, dag_ret:0, dag_status:2, start_time:1727337655668059, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3490, output_block_cnt_arr:0x7f92728f3498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337655668072, estimated_finish_time:1727337672673771}) [2024-09-26 16:00:55.673757] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:333}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.673779] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.673778] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9272830c30, type:15, status:2, dag:{ObIDag:{this:0x7f9272830870, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E2-0-0, dag_ret:0, dag_status:2, start_time:1727337655668059, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.673783] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.673783] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=5] task finish process(ret=0, start_time=1727337655668071, end_time=1727337655673783, runtime=5712, *this={this:0x7f9272830c30, type:15, status:2, dag:{ObIDag:{this:0x7f9272830870, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E2-0-0, dag_ret:0, dag_status:2, start_time:1727337655668059, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.673785] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=22] succeed to build merge ctx(tablet_id={id:333}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:333}, table_count:1, [{i:0, table_key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8651da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655671960, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.79ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.673806] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=3] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.673789] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101009}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:00:55.673822] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=23] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92728331f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EA-0-0, dag_ret:0, dag_status:2, start_time:1727337655671623, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b490, output_block_cnt_arr:0x7f9263a4b498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337655671960, estimated_finish_time:1727337672673822}) [2024-09-26 16:00:55.673831] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f92728335b0, type:15, status:2, dag:{ObIDag:{this:0x7f92728331f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EA-0-0, dag_ret:0, dag_status:2, start_time:1727337655671623, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.673819] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=24] succeed to build merge ctx(tablet_id={id:101009}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101009}, table_count:1, [{i:0, table_key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8641da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655670041, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.75ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.673838] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=7] task finish process(ret=0, start_time=1727337655671959, end_time=1727337655673838, runtime=1879, *this={this:0x7f92728335b0, type:15, status:2, dag:{ObIDag:{this:0x7f92728331f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EA-0-0, dag_ret:0, dag_status:2, start_time:1727337655671623, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.673847] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92728312d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E4-0-0, dag_ret:0, dag_status:2, start_time:1727337655669846, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a45470, output_block_cnt_arr:0x7f9263a45478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337655670041, estimated_finish_time:1727337672673846}) [2024-09-26 16:00:55.673854] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9272831690, type:15, status:2, dag:{ObIDag:{this:0x7f92728312d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E4-0-0, dag_ret:0, dag_status:2, start_time:1727337655669846, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.673859] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=5] task finish process(ret=0, start_time=1727337655670039, end_time=1727337655673859, runtime=3820, *this={this:0x7f9272831690, type:15, status:2, dag:{ObIDag:{this:0x7f92728312d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E4-0-0, dag_ret:0, dag_status:2, start_time:1727337655669846, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.673844] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=342, range=0x7f92ab056200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:21, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:23}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685e6060, count:6, allocator:0x7f92636aa7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:23, datum_buffer:{capacity:32, datums:0x7f92685e6390, local_datums:0x7f92685e6390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,]}) [2024-09-26 16:00:55.673872] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337655632011, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.673884] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=23][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.673884] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752949376, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=20, max_schema_version_in_memtable=1710828752949376, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.673904] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.673907] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.673922] ERROR issue_dba_error (ob_log.cpp:1875) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.673913] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:323}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:22, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752949376, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.673925] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.673919] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101066, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:7}, cols_desc:{ObIArray:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d32060, count:7, allocator:0x7f92c1d9a7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9287d323b0, local_datums:0x7f9287d323b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:00:55.673935] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337655631954, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.673935] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:323}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752949376, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:323}, table_count:1, [{i:0, table_key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655670638, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.27ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.673955] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655673817, end_time=1727337655673955, runtime=138, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272830870, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E2-0-0, dag_ret:0, dag_status:2, start_time:1727337655668059, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.673965] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5905, dag_cnt=21, dag_cnts_[dag.get_type()]=21, &dag=0x7f9272830870, dag={ObIDag:{this:0x7f9272830870, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E2-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655668059, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.673972] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E2-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655668055, task_id:YB42C0A8002D-0006230056D8C6E2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.673969] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272831d30, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E6-0-0, dag_ret:0, dag_status:2, start_time:1727337655670273, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db9480, output_block_cnt_arr:0x7f9287db9488, concurrent_cnt:1, estimate_row_cnt:205, estimate_occupy_size:0, latest_update_ts:1727337655670638, estimated_finish_time:1727337672673969}) [2024-09-26 16:00:55.673977] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f92728320f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272831d30, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E6-0-0, dag_ret:0, dag_status:2, start_time:1727337655670273, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.673984] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=7] task finish process(ret=0, start_time=1727337655670636, end_time=1727337655673984, runtime=3348, *this={this:0x7f92728320f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272831d30, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E6-0-0, dag_ret:0, dag_status:2, start_time:1727337655670273, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.673990] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=3] schedule one task(task=0x7f92729263c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.673996] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=3] schedule one task(task=0x7f9264c3a3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.674000] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655673999, task_id:YB42C0A8002D-0006230056D8C6EC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.674003] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=3] schedule one task(task=0x7f9272834010, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.674075] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.674068] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=333, range=0x7f92ab0d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:7, allocator:0x7f92bc9147d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9263b5e3b0, local_datums:0x7f9263b5e3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:55.674114] ERROR issue_dba_error (ob_log.cpp:1875) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.674117] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.674128] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=0] partition merge iter row count(i=0, row_count=182, ghost_row_count=0, pkey={tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337655632084, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.674151] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337655673889, end_time=1727337655674151, runtime=262, *this={this:0x7f9272926030, type:1, status:2, dag:{ObIDag:{this:0x7f9272833720, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EB-0-0, dag_ret:0, dag_status:2, start_time:1727337655671817, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.674162] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2344, dag_cnt=20, dag_cnts_[dag.get_type()]=20, &dag=0x7f9272833720, dag={ObIDag:{this:0x7f9272833720, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EB-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655671817, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.674170] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EB-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655671813, task_id:YB42C0A8002D-0006230056D8C6EB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.674186] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655674184, task_id:YB42C0A8002D-0006230056D8C6ED-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.674189] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=3] schedule one task(task=0x7f9272834540, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.674223] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=101009, range=0x7f92a9952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ec060, count:4, allocator:0x7f92b321a7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92664ec350, local_datums:0x7f92664ec350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:00:55.674256] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.674284] ERROR issue_dba_error (ob_log.cpp:1875) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.674287] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.674294] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337655631982, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.674310] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655674196, end_time=1727337655674310, runtime=114, *this={this:0x7f92729263c0, type:1, status:2, dag:{ObIDag:{this:0x7f92728312d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E4-0-0, dag_ret:0, dag_status:2, start_time:1727337655669846, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.674318] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4472, dag_cnt=19, dag_cnts_[dag.get_type()]=19, &dag=0x7f92728312d0, dag={ObIDag:{this:0x7f92728312d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E4-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655669846, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.674325] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E4-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655669843, task_id:YB42C0A8002D-0006230056D8C6E4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.674329] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=22][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.674336] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655674335, task_id:YB42C0A8002D-0006230056D8C6EE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.674347] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=10] schedule one task(task=0x7f9272834a70, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.674410] ERROR issue_dba_error (ob_log.cpp:1875) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=8][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.674415] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.674428] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=0] partition merge iter row count(i=0, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337655632069, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.674455] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655674025, end_time=1727337655674455, runtime=430, *this={this:0x7f9264c3a3c0, type:1, status:2, dag:{ObIDag:{this:0x7f92728331f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EA-0-0, dag_ret:0, dag_status:2, start_time:1727337655671623, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.674487] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2864, dag_cnt=18, dag_cnts_[dag.get_type()]=18, &dag=0x7f92728331f0, dag={ObIDag:{this:0x7f92728331f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EA-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655671623, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.674498] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EA-0-0] [lt=9] succeed to del sys task(removed_task={start_time:1727337655671618, task_id:YB42C0A8002D-0006230056D8C6EA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.674516] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655674514, task_id:YB42C0A8002D-0006230056D8C6EF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.674519] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=2] schedule one task(task=0x7f9272834fa0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.674540] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.674544] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.674547] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:55.674552] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.674557] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077691154184001, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8650e00, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:497, tablet_addr:[809012-1445-0-2305843637353218048-2](offset=557056,size=585,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:451}, data_tablet_id:{id:451}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077691154184001, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2119849913, row_count:31}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86514b8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844404004851712-0](offset=528384,size=1299,type=2,seq=0), type:"MAJOR", tablet_id:{id:451}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563069147000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843980950574355-0](offset=529683,size=905,type=2,seq=0), type:"MINI", tablet_id:{id:451}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986382094000, upper_trans_version:1726149986382094000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337655632113, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdfc00, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334222657108}, this:0x7f92a3bdfc00, timestamp:1727334222657108, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753331112, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:89, clock:641728512}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293246986000, v:0}, rec_scn:{val:1726265293246986000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696516096-2](offset=536576,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843170275504460-0](offset=536908,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879883234-0](offset=537058,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446403268-0](offset=537284,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc4080, 0x7f92a3bdfc00, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.674638] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=80] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.674647] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.674732] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.674736] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.674739] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:55.674744] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.674748] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.674755] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.674788] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.674809] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=21][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.674812] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:55.674816] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.674820] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.674841] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.674884] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337655632113, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.674908] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=23] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9264a69330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=14, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.674922] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.674927] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.674934] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9264a69330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:451}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:16, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.674956] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:451}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9264a69330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:451}, table_count:1, [{i:0, table_key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8650e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655674534, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=397us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.674998] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272834180, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6ED-0-0, dag_ret:0, dag_status:2, start_time:1727337655674189, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264a69470, output_block_cnt_arr:0x7f9264a69478, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337655674534, estimated_finish_time:1727337672674997}) [2024-09-26 16:00:55.675008] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9272834540, type:15, status:2, dag:{ObIDag:{this:0x7f9272834180, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6ED-0-0, dag_ret:0, dag_status:2, start_time:1727337655674189, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.675016] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=7] task finish process(ret=0, start_time=1727337655674532, end_time=1727337655675015, runtime=483, *this={this:0x7f9272834540, type:15, status:2, dag:{ObIDag:{this:0x7f9272834180, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6ED-0-0, dag_ret:0, dag_status:2, start_time:1727337655674189, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.675036] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=1] schedule one task(task=0x7f9264b46030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.675062] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=3] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.675072] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337655632025, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.675085] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=41, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.675097] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.675101] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.675107] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:377}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:43, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.675130] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=18] succeed to build merge ctx(tablet_id={id:377}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:377}, table_count:1, [{i:0, table_key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655670733, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.37ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.675155] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272832260, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E7-0-0, dag_ret:0, dag_status:2, start_time:1727337655670498, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515480, output_block_cnt_arr:0x7f9268515488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337655670733, estimated_finish_time:1727337672675155}) [2024-09-26 16:00:55.675163] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9272832620, type:15, status:2, dag:{ObIDag:{this:0x7f9272832260, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E7-0-0, dag_ret:0, dag_status:2, start_time:1727337655670498, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.675170] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=7] task finish process(ret=0, start_time=1727337655670732, end_time=1727337655675170, runtime=4438, *this={this:0x7f9272832620, type:15, status:2, dag:{ObIDag:{this:0x7f9272832260, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E7-0-0, dag_ret:0, dag_status:2, start_time:1727337655670498, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.675189] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=2] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.675258] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=377, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:41, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:43}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fa060, count:5, allocator:0x7f92864e07d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:43, datum_buffer:{capacity:43, datums:0x7f92665fb2b0, local_datums:0x7f92665fa360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,]}) [2024-09-26 16:00:55.675332] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=23][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.675407] ERROR issue_dba_error (ob_log.cpp:1875) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.675412] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.675424] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=0] partition merge iter row count(i=0, row_count=6, ghost_row_count=0, pkey={tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337655632025, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.675482] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655675202, end_time=1727337655675481, runtime=279, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9272832260, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E7-0-0, dag_ret:0, dag_status:2, start_time:1727337655670498, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.675495] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=1] dag finished(dag_ret=-4184, runtime=4997, dag_cnt=17, dag_cnts_[dag.get_type()]=17, &dag=0x7f9272832260, dag={ObIDag:{this:0x7f9272832260, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E7-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655670498, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.675507] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E7-0-0] [lt=9] succeed to del sys task(removed_task={start_time:1727337655670494, task_id:YB42C0A8002D-0006230056D8C6E7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.675504] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=8] multi version scan iterator init succ(param.table_id_=323, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:20, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:22}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:5, allocator:0x7f92c1d9e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,]}) [2024-09-26 16:00:55.675524] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337655675522, task_id:YB42C0A8002D-0006230056D8C6F0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.675527] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=3] schedule one task(task=0x7f92728354d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.675540] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337655632128, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.675552] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.675567] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.675571] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.675577] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:378}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.675600] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=18] succeed to build merge ctx(tablet_id={id:378}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:378}, table_count:1, [{i:0, table_key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655674727, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=848us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.675626] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92728346b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EE-0-0, dag_ret:0, dag_status:2, start_time:1727337655674347, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b480, output_block_cnt_arr:0x7f9263a4b488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337655674727, estimated_finish_time:1727337672675626}) [2024-09-26 16:00:55.675635] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9272834a70, type:15, status:2, dag:{ObIDag:{this:0x7f92728346b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EE-0-0, dag_ret:0, dag_status:2, start_time:1727337655674347, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.675642] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=7] task finish process(ret=0, start_time=1727337655674725, end_time=1727337655675642, runtime=917, *this={this:0x7f9272834a70, type:15, status:2, dag:{ObIDag:{this:0x7f92728346b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EE-0-0, dag_ret:0, dag_status:2, start_time:1727337655674347, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.675670] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=2] schedule one task(task=0x7f9264d76030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.675686] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337655632099, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.675698] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=19, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.675722] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=378, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92729f8060, count:5, allocator:0x7f92a31f47d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92729f8360, local_datums:0x7f92729f8360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:55.675749] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.675749] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.675752] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.675758] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:334}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:21, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.675779] ERROR issue_dba_error (ob_log.cpp:1875) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.675781] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.675785] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.675789] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.675790] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337655632128, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.675783] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=20] succeed to build merge ctx(tablet_id={id:334}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:334}, table_count:1, [{i:0, table_key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8654c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655674784, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=972us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.675798] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=0] partition merge iter row count(i=0, row_count=171, ghost_row_count=0, pkey={tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337655632011, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.675810] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655675681, end_time=1727337655675810, runtime=129, *this={this:0x7f9264d76030, type:1, status:2, dag:{ObIDag:{this:0x7f92728346b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EE-0-0, dag_ret:0, dag_status:2, start_time:1727337655674347, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.675819] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1472, dag_cnt=16, dag_cnts_[dag.get_type()]=16, &dag=0x7f92728346b0, dag={ObIDag:{this:0x7f92728346b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EE-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655674347, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.675819] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655675451, end_time=1727337655675819, runtime=368, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9272831d30, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E6-0-0, dag_ret:0, dag_status:2, start_time:1727337655670273, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.675826] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EE-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655674335, task_id:YB42C0A8002D-0006230056D8C6EE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.675849] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337655675848, task_id:YB42C0A8002D-0006230056D8C6F1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.675851] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=2] schedule one task(task=0x7f9272835a00, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.675848] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=22] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272833c50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EC-0-0, dag_ret:0, dag_status:2, start_time:1727337655674003, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab4a0, output_block_cnt_arr:0x7f92684ab4a8, concurrent_cnt:1, estimate_row_cnt:993, estimate_occupy_size:0, latest_update_ts:1727337655674784, estimated_finish_time:1727337672675847}) [2024-09-26 16:00:55.675866] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=18] succeed to init merge ctx(task={this:0x7f9272834010, type:15, status:2, dag:{ObIDag:{this:0x7f9272833c50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EC-0-0, dag_ret:0, dag_status:2, start_time:1727337655674003, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.675876] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.675874] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=8] task finish process(ret=0, start_time=1727337655674783, end_time=1727337655675874, runtime=1091, *this={this:0x7f9272834010, type:15, status:2, dag:{ObIDag:{this:0x7f9272833c50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EC-0-0, dag_ret:0, dag_status:2, start_time:1727337655674003, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.675880] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.675882] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:55.675887] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.675896] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=2] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.675907] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.675912] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.675915] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:55.675920] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.675891] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b861fe60, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:491, tablet_addr:[816017-1849-0-2305843630912028672-2](offset=1818624,size=579,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:215}, data_tablet_id:{id:215}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3095927781, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86204f4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844118390796288-0](offset=1798144,size=1033,type=2,seq=0), type:"MAJOR", tablet_id:{id:215}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389038459000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843910084883465-0](offset=1799177,size=839,type=2,seq=0), type:"MINOR", tablet_id:{id:215}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726209400113641000, upper_trans_version:1726209400113641000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337655632156, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcf430, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209475373}, this:0x7f92a3bcf430, timestamp:1727334209475373, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:56, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277800266460000, v:0}, rec_scn:{val:1726241800098150000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697789952-2](offset=1810432,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834327372-0](offset=1810764,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881157084-0](offset=1810908,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447677118-0](offset=1811134,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc86d0, 0x7f92a3bcf430, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.675939] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.675943] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=52] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.675949] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.675950] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.675952] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:55.675957] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.676016] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=8] multi version scan iterator init succ(param.table_id_=334, range=0x7f92aa352200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:19, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:21}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d30060, count:8, allocator:0x7f92b32387d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:21, datum_buffer:{capacity:32, datums:0x7f9287d303e0, local_datums:0x7f9287d303e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,]}) [2024-09-26 16:00:55.676158] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337655632170, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.676170] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753355744, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753355744, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.676182] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.676186] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.676192] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:456}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753355744, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.676210] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:456}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753355744, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:456}, table_count:1, [{i:0, table_key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8634c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655675902, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=287us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.676236] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.676243] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272835640, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F1-0-0, dag_ret:0, dag_status:2, start_time:1727337655675851, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b490, output_block_cnt_arr:0x7f9263a4b498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337655675902, estimated_finish_time:1727337672676242}) [2024-09-26 16:00:55.676250] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9272835a00, type:15, status:2, dag:{ObIDag:{this:0x7f9272835640, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F1-0-0, dag_ret:0, dag_status:2, start_time:1727337655675851, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.676255] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=5] task finish process(ret=0, start_time=1727337655675900, end_time=1727337655676255, runtime=355, *this={this:0x7f9272835a00, type:15, status:2, dag:{ObIDag:{this:0x7f9272835640, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F1-0-0, dag_ret:0, dag_status:2, start_time:1727337655675851, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.676271] ERROR issue_dba_error (ob_log.cpp:1875) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.676276] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.676274] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=3] schedule one task(task=0x7f9287c663c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.676286] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=0] partition merge iter row count(i=0, row_count=153, ghost_row_count=0, pkey={tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337655632099, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.676304] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655675989, end_time=1727337655676304, runtime=315, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9272833c50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EC-0-0, dag_ret:0, dag_status:2, start_time:1727337655674003, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.676314] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2311, dag_cnt=15, dag_cnts_[dag.get_type()]=15, &dag=0x7f9272833c50, dag={ObIDag:{this:0x7f9272833c50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EC-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655674003, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.676330] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EC-0-0] [lt=13] succeed to del sys task(removed_task={start_time:1727337655673999, task_id:YB42C0A8002D-0006230056D8C6EC-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.676326] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=456, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684ac060, count:7, allocator:0x7f92865547d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f92684ac3b0, local_datums:0x7f92684ac3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:00:55.676364] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=6090, dag_cnt=14, dag_cnts_[dag.get_type()]=14, &dag=0x7f9272831d30, dag={ObIDag:{this:0x7f9272831d30, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6E6-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655670273, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.676371] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6E6-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655670268, task_id:YB42C0A8002D-0006230056D8C6E6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.676430] ERROR issue_dba_error (ob_log.cpp:1875) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.676433] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.676433] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=451, range=0x7f92aa952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:14, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:16}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665ae060, count:4, allocator:0x7f92b32407d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:16, datum_buffer:{capacity:32, datums:0x7f92665ae350, local_datums:0x7f92665ae350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,]}) [2024-09-26 16:00:55.676441] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=0] partition merge iter row count(i=0, row_count=45, ghost_row_count=0, pkey={tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337655632170, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.676456] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.676461] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655676293, end_time=1727337655676461, runtime=168, *this={this:0x7f9287c663c0, type:1, status:2, dag:{ObIDag:{this:0x7f9272835640, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F1-0-0, dag_ret:0, dag_status:2, start_time:1727337655675851, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.676486] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=635, dag_cnt=13, dag_cnts_[dag.get_type()]=13, &dag=0x7f9272835640, dag={ObIDag:{this:0x7f9272835640, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F1-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655675851, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.676493] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F1-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655675848, task_id:YB42C0A8002D-0006230056D8C6F1-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.676511] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] succeed to add sys task(task={start_time:1727337655676510, task_id:YB42C0A8002D-0006230056D8C6F2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.676513] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=2] schedule one task(task=0x7f9272835f30, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.676519] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655676518, task_id:YB42C0A8002D-0006230056D8C6F3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.676521] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=2] schedule one task(task=0x7f9272836460, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.676525] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655676524, task_id:YB42C0A8002D-0006230056D8C6F4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.676516] ERROR issue_dba_error (ob_log.cpp:1875) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.676527] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=1] schedule one task(task=0x7f9272836990, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.676528] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=12][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.676542] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.676537] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337655632113, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.676546] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.676548] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:55.676552] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.676556] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655676398, end_time=1727337655676556, runtime=158, *this={this:0x7f9264b46030, type:1, status:2, dag:{ObIDag:{this:0x7f9272834180, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6ED-0-0, dag_ret:0, dag_status:2, start_time:1727337655674189, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.676565] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2376, dag_cnt=12, dag_cnts_[dag.get_type()]=12, &dag=0x7f9272834180, dag={ObIDag:{this:0x7f9272834180, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6ED-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655674189, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.676572] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6ED-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655674184, task_id:YB42C0A8002D-0006230056D8C6ED-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.676555] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b863eec0, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:499, tablet_addr:[816017-1849-0-2305843639502065664-2](offset=1921024,size=587,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101016}, data_tablet_id:{id:154}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726074677203981000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:398898191, row_count:578}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844419038609408-0](offset=1900544,size=1313,type=2,seq=0), type:"MAJOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726074015354550000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844024975361313-0](offset=1901857,size=946,type=2,seq=0), type:"MINOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:1726074677203981000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726205802467242000, upper_trans_version:1726205802467242000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337655632214, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd5fc0, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209508808}, this:0x7f92a3bd5fc0, timestamp:1727334209508808, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:69, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277803107894000, v:0}, rec_scn:{val:1726241804194470000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771625984-2](offset=1904640,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161774413-0](offset=1904973,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881251311-0](offset=1905135,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc7f00, 0x7f92a3bd5fc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.676613] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=57] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.676613] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=7][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.676616] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.676618] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:55.676619] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.676622] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.676625] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.676641] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=12] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.676642] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337655676640, task_id:YB42C0A8002D-0006230056D8C6F5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.676645] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=2] schedule one task(task=0x7f9272836ec0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.676684] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.676687] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.676688] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:55.676694] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.676697] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.676701] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.676713] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337655632156, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.676722] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752725472, storage_schema:{this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752725472, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.676734] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.676737] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.676743] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:215}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752725472, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.676759] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:215}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752725472, storage_schema:{this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:215}, table_count:1, [{i:0, table_key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655675870, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=871us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.676796] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272835110, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F0-0-0, dag_ret:0, dag_status:2, start_time:1727337655675527, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92664ed460, output_block_cnt_arr:0x7f92664ed468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337655675870, estimated_finish_time:1727337672676796}) [2024-09-26 16:00:55.676804] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f92728354d0, type:15, status:2, dag:{ObIDag:{this:0x7f9272835110, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F0-0-0, dag_ret:0, dag_status:2, start_time:1727337655675527, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.676809] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=5] task finish process(ret=0, start_time=1727337655675868, end_time=1727337655676809, runtime=941, *this={this:0x7f92728354d0, type:15, status:2, dag:{ObIDag:{this:0x7f9272835110, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F0-0-0, dag_ret:0, dag_status:2, start_time:1727337655675527, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.676851] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=10][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.676856] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.676858] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:55.676862] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.676865] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.676869] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.676880] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337655632199, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.676852] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337655632214, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.676891] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=38] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.676888] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.676904] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.676907] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=16] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.676907] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.676910] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.676912] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101016}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.676914] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101015}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.676930] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101015}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101015}, table_count:1, [{i:0, table_key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655676608, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=305us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.676929] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:101016}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101016}, table_count:1, [{i:0, table_key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655676537, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=373us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.676949] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92728360a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F3-0-0, dag_ret:0, dag_status:2, start_time:1727337655676521, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b490, output_block_cnt_arr:0x7f9263a4b498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337655676608, estimated_finish_time:1727337672676949}) [2024-09-26 16:00:55.676953] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92728365d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F4-0-0, dag_ret:0, dag_status:2, start_time:1727337655676526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad490, output_block_cnt_arr:0x7f92684ad498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337655676537, estimated_finish_time:1727337672676952}) [2024-09-26 16:00:55.676955] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9272836460, type:15, status:2, dag:{ObIDag:{this:0x7f92728360a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F3-0-0, dag_ret:0, dag_status:2, start_time:1727337655676521, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.676958] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272836990, type:15, status:2, dag:{ObIDag:{this:0x7f92728365d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F4-0-0, dag_ret:0, dag_status:2, start_time:1727337655676526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.676959] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=4] task finish process(ret=0, start_time=1727337655676606, end_time=1727337655676959, runtime=353, *this={this:0x7f9272836460, type:15, status:2, dag:{ObIDag:{this:0x7f92728360a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F3-0-0, dag_ret:0, dag_status:2, start_time:1727337655676521, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.676963] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=5] task finish process(ret=0, start_time=1727337655676536, end_time=1727337655676963, runtime=427, *this={this:0x7f9272836990, type:15, status:2, dag:{ObIDag:{this:0x7f92728365d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F4-0-0, dag_ret:0, dag_status:2, start_time:1727337655676526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.676976] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=2] schedule one task(task=0x7f9268486030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.676981] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=3] schedule one task(task=0x7f9272838f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.676986] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=4] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.676995] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337655632185, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.677004] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.677012] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.677014] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.677018] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:154}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.677042] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=20] succeed to build merge ctx(tablet_id={id:154}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:154}, table_count:1, [{i:0, table_key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8623ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655676680, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=336us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.677062] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272835b70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F2-0-0, dag_ret:0, dag_status:2, start_time:1727337655676513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515480, output_block_cnt_arr:0x7f9268515488, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337655676680, estimated_finish_time:1727337672677062}) [2024-09-26 16:00:55.677067] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272835f30, type:15, status:2, dag:{ObIDag:{this:0x7f9272835b70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F2-0-0, dag_ret:0, dag_status:2, start_time:1727337655676513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.677072] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=5] task finish process(ret=0, start_time=1727337655676678, end_time=1727337655677072, runtime=394, *this={this:0x7f9272835f30, type:15, status:2, dag:{ObIDag:{this:0x7f9272835b70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F2-0-0, dag_ret:0, dag_status:2, start_time:1727337655676513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.677088] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=2] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.677123] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=7] multi version scan iterator init succ(param.table_id_=101016, range=0x7f92aac56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d32060, count:6, allocator:0x7f926361a7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9287d32390, local_datums:0x7f9287d32390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:55.677167] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.677200] ERROR issue_dba_error (ob_log.cpp:1875) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.677203] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.677211] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337655632214, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.677227] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655677097, end_time=1727337655677227, runtime=130, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f92728365d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F4-0-0, dag_ret:0, dag_status:2, start_time:1727337655676526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.677235] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=709, dag_cnt=11, dag_cnts_[dag.get_type()]=11, &dag=0x7f92728365d0, dag={ObIDag:{this:0x7f92728365d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F4-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655676526, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.677244] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F4-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337655676524, task_id:YB42C0A8002D-0006230056D8C6F4-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.677257] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337655677255, task_id:YB42C0A8002D-0006230056D8C6F6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.677260] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=3] schedule one task(task=0x7f92728373f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.677276] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.677279] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.677281] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:55.677285] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.677288] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.677293] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.677291] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=154, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:5, allocator:0x7f92636927d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a62360, local_datums:0x7f9264a62360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:00:55.677367] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=29][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.677399] ERROR issue_dba_error (ob_log.cpp:1875) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.677402] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.677410] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337655632185, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.677427] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655677265, end_time=1727337655677427, runtime=162, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9272835b70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F2-0-0, dag_ret:0, dag_status:2, start_time:1727337655676513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.677436] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=923, dag_cnt=10, dag_cnts_[dag.get_type()]=10, &dag=0x7f9272835b70, dag={ObIDag:{this:0x7f9272835b70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F2-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655676513, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.677443] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F2-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655676510, task_id:YB42C0A8002D-0006230056D8C6F2-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.677456] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655677454, task_id:YB42C0A8002D-0006230056D8C6F7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.677459] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=3] schedule one task(task=0x7f9272837920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.677474] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.677477] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.677479] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:55.677482] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.677485] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706139416032, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f926e072bc0, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[812168-2208-0-2305843644869693440-2](offset=839680,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:328}, data_tablet_id:{id:328}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706139416032, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:867269498, row_count:786}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f926e07329c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[812168-2208-0-2305844589762490368-0](offset=831488,size=1472,type=2,seq=0), type:"MAJOR", tablet_id:{id:328}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562828712000}}, minor_tables:{{i:0, addr:[812168-2208-0-2305843935853721024-0](offset=832960,size=863,type=2,seq=0), type:"MINI", tablet_id:{id:328}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986144852000, upper_trans_version:1726149986144852000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337655632258, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bda610, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220621826}, this:0x7f92a3bda610, timestamp:1727334220621826, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:78, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265257450254001, v:0}, rec_scn:{val:1726264907253071001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[812168-2208-0-2305843366770556928-2](offset=835584,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[812168-2208-0-2305843176718254413-0](offset=835917,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843251880182249-0](offset=836073,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843216446702283-0](offset=836299,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc8ea0, 0x7f92a3bda610, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.677535] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=50] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.677536] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=7] multi version scan iterator init succ(param.table_id_=101015, range=0x7f92a9cd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:6, allocator:0x7f92bc9147d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92665f8390, local_datums:0x7f92665f8390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:55.677540] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.677600] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.677632] ERROR issue_dba_error (ob_log.cpp:1875) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.677635] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.677644] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337655632199, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.677662] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655677501, end_time=1727337655677662, runtime=161, *this={this:0x7f9272838f50, type:1, status:2, dag:{ObIDag:{this:0x7f92728360a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F3-0-0, dag_ret:0, dag_status:2, start_time:1727337655676521, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.677670] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1149, dag_cnt=9, dag_cnts_[dag.get_type()]=9, &dag=0x7f92728360a0, dag={ObIDag:{this:0x7f92728360a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F3-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655676521, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.677677] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F3-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655676518, task_id:YB42C0A8002D-0006230056D8C6F3-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.677691] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655677689, task_id:YB42C0A8002D-0006230056D8C6F8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.677694] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=3] schedule one task(task=0x7f9272837e50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.677708] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337655632243, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.677716] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=15, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.677727] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.677729] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.677733] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.677736] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.677738] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:55.677742] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.677735] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:452}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:17, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.677746] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.677752] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.677751] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:452}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:452}, table_count:1, [{i:0, table_key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8652d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655677272, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=461us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.677788] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=31] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272837030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F6-0-0, dag_ret:0, dag_status:2, start_time:1727337655677260, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684b1480, output_block_cnt_arr:0x7f92684b1488, concurrent_cnt:1, estimate_row_cnt:10, estimate_occupy_size:0, latest_update_ts:1727337655677272, estimated_finish_time:1727337672677787}) [2024-09-26 16:00:55.677793] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f92728373f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272837030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F6-0-0, dag_ret:0, dag_status:2, start_time:1727337655677260, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.677798] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=5] task finish process(ret=0, start_time=1727337655677270, end_time=1727337655677798, runtime=528, *this={this:0x7f92728373f0, type:15, status:2, dag:{ObIDag:{this:0x7f9272837030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F6-0-0, dag_ret:0, dag_status:2, start_time:1727337655677260, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.677812] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=215, range=0x7f92aaed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:6}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d34060, count:3, allocator:0x7f92bc9147d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9287d34320, local_datums:0x7f9287d34320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:00:55.677839] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=2] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.677896] ERROR issue_dba_error (ob_log.cpp:1875) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.677899] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.677908] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=0] partition merge iter row count(i=0, row_count=9, ghost_row_count=0, pkey={tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337655632156, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.677925] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655677788, end_time=1727337655677925, runtime=137, *this={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9272835110, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F0-0-0, dag_ret:0, dag_status:2, start_time:1727337655675527, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.677935] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=1] dag finished(dag_ret=-4184, runtime=2408, dag_cnt=8, dag_cnts_[dag.get_type()]=8, &dag=0x7f9272835110, dag={ObIDag:{this:0x7f9272835110, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F0-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655675527, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.677941] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F0-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337655675522, task_id:YB42C0A8002D-0006230056D8C6F0-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.677962] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655677960, task_id:YB42C0A8002D-0006230056D8C6F9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.677965] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=3] schedule one task(task=0x7f9272838380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.678023] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.678026] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.678030] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:00:55.678034] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.678037] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.678034] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=25] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337655632258, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.678043] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.678044] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=9] update column cnt(column_cnt_=26, store_column_cnt_=26, column_cnt_=26, column_array_.count()=0) [2024-09-26 16:00:55.678047] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9287db7480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=26, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.678057] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.678060] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.678066] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:328}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:28, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.678081] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=452, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:15, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:17}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a66060, count:5, allocator:0x7f92636967d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:17, datum_buffer:{capacity:32, datums:0x7f9264a66360, local_datums:0x7f9264a66360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,]}) [2024-09-26 16:00:55.678084] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:328}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9287db7480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:328}, table_count:1, [{i:0, table_key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e072bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655677469, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=593us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.678104] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272837560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F7-0-0, dag_ret:0, dag_status:2, start_time:1727337655677459, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db75d0, output_block_cnt_arr:0x7f9287db75d8, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337655677469, estimated_finish_time:1727337672678104}) [2024-09-26 16:00:55.678110] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272837920, type:15, status:2, dag:{ObIDag:{this:0x7f9272837560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F7-0-0, dag_ret:0, dag_status:2, start_time:1727337655677459, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678114] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=4] task finish process(ret=0, start_time=1727337655677468, end_time=1727337655678114, runtime=646, *this={this:0x7f9272837920, type:15, status:2, dag:{ObIDag:{this:0x7f9272837560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F7-0-0, dag_ret:0, dag_status:2, start_time:1727337655677459, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678117] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.678135] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=2] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.678160] ERROR issue_dba_error (ob_log.cpp:1875) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.678163] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.678171] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=0] partition merge iter row count(i=0, row_count=10, ghost_row_count=0, pkey={tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337655632243, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.678187] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655678055, end_time=1727337655678187, runtime=132, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9272837030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F6-0-0, dag_ret:0, dag_status:2, start_time:1727337655677260, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678196] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=936, dag_cnt=7, dag_cnts_[dag.get_type()]=7, &dag=0x7f9272837030, dag={ObIDag:{this:0x7f9272837030, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F6-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655677260, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.678203] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F6-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655677255, task_id:YB42C0A8002D-0006230056D8C6F6-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.678190] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=328, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:26, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:28}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:5, allocator:0x7f92865b67d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:28, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,]}) [2024-09-26 16:00:55.678215] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655678213, task_id:YB42C0A8002D-0006230056D8C6FA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.678218] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=3] schedule one task(task=0x7f92728388b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.678224] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337655632275, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.678233] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=8] update column cnt(column_cnt_=12, store_column_cnt_=12, column_cnt_=12, column_array_.count()=0) [2024-09-26 16:00:55.678236] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9263a4b490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.678255] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.678258] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.678269] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.678263] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:330}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.678283] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=13][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.678292] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337655632258, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.678294] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=26] succeed to build merge ctx(tablet_id={id:330}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9263a4b490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:330}, table_count:1, [{i:0, table_key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e107fe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655677727, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=534us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.678313] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655678147, end_time=1727337655678313, runtime=166, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9272837560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F7-0-0, dag_ret:0, dag_status:2, start_time:1727337655677459, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678323] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=864, dag_cnt=6, dag_cnts_[dag.get_type()]=6, &dag=0x7f9272837560, dag={ObIDag:{this:0x7f9272837560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F7-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655677459, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.678325] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272837a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F8-0-0, dag_ret:0, dag_status:2, start_time:1727337655677694, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b5f0, output_block_cnt_arr:0x7f9263a4b5f8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337655677727, estimated_finish_time:1727337672678324}) [2024-09-26 16:00:55.678331] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F7-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655677454, task_id:YB42C0A8002D-0006230056D8C6F7-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.678330] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272837e50, type:15, status:2, dag:{ObIDag:{this:0x7f9272837a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F8-0-0, dag_ret:0, dag_status:2, start_time:1727337655677694, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678335] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=5] task finish process(ret=0, start_time=1727337655677726, end_time=1727337655678335, runtime=609, *this={this:0x7f9272837e50, type:15, status:2, dag:{ObIDag:{this:0x7f9272837a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F8-0-0, dag_ret:0, dag_status:2, start_time:1727337655677694, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678349] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337655678348, task_id:YB42C0A8002D-0006230056D8C6FB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.678352] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=3] schedule one task(task=0x7f9272838de0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.678370] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.678371] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.678373] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.678374] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.678375] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:55.678376] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:55.678372] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337655632291, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.678379] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.678380] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.678381] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=8] update column cnt(column_cnt_=29, store_column_cnt_=29, column_cnt_=29, column_array_.count()=0) [2024-09-26 16:00:55.678383] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.678383] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9287db9490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.678389] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.678392] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.678395] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.678382] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b86a2d40, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:511, tablet_addr:[816017-1849-0-2305843652387110912-2](offset=2064384,size=599,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101067}, data_tablet_id:{id:334}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1854554341, row_count:13383}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86a3488, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844595132416000-0](offset=2048000,size=1477,type=2,seq=0), type:"MAJOR", tablet_id:{id:101067}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843951961064901-0](offset=2049477,size=878,type=2,seq=0), type:"MINI", tablet_id:{id:101067}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}, {i:1, addr:[816017-1849-0-2305843958403516723-0](offset=2050355,size=884,type=2,seq=0), type:"MINI", tablet_id:{id:101067}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726208018709503000, upper_trans_version:1726208018709503000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337655632345, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bde490, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221276956}, this:0x7f92a3bde490, timestamp:1727334221276956, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:86, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265292880560000, v:0}, rec_scn:{val:1726264918599518000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843482735894528-2](offset=2056192,size=441,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046827961-0](offset=2056633,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881402983-0](offset=2056807,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bcbd80, 0x7f92a3bde490, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.678408] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=11] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:329}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.678419] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=37] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.678425] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.678425] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:329}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9287db9490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:329}, table_count:1, [{i:0, table_key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e089f20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655678019, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=387us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.678464] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272837fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F9-0-0, dag_ret:0, dag_status:2, start_time:1727337655677965, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db95f0, output_block_cnt_arr:0x7f9287db95f8, concurrent_cnt:1, estimate_row_cnt:62, estimate_occupy_size:0, latest_update_ts:1727337655678019, estimated_finish_time:1727337672678464}) [2024-09-26 16:00:55.678471] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9272838380, type:15, status:2, dag:{ObIDag:{this:0x7f9272837fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F9-0-0, dag_ret:0, dag_status:2, start_time:1727337655677965, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678476] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=5] task finish process(ret=0, start_time=1727337655678017, end_time=1727337655678476, runtime=459, *this={this:0x7f9272838380, type:15, status:2, dag:{ObIDag:{this:0x7f9272837fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F9-0-0, dag_ret:0, dag_status:2, start_time:1727337655677965, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678494] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=2] schedule one task(task=0x7f9272838f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.678499] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=4] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.678522] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337655632228, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.678531] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.678541] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.678544] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.678549] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101063}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.678539] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=330, range=0x7f92aab52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d30060, count:7, allocator:0x7f9269ffa7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9287d303b0, local_datums:0x7f9287d303b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:00:55.678566] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:101063}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101063}, table_count:1, [{i:0, table_key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655676840, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.71ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.678587] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272836b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F5-0-0, dag_ret:0, dag_status:2, start_time:1727337655676645, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db54a0, output_block_cnt_arr:0x7f9287db54a8, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337655676840, estimated_finish_time:1727337672678587}) [2024-09-26 16:00:55.678592] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9272836ec0, type:15, status:2, dag:{ObIDag:{this:0x7f9272836b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F5-0-0, dag_ret:0, dag_status:2, start_time:1727337655676645, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678599] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=7] task finish process(ret=0, start_time=1727337655676839, end_time=1727337655678599, runtime=1760, *this={this:0x7f9272836ec0, type:15, status:2, dag:{ObIDag:{this:0x7f9272836b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F5-0-0, dag_ret:0, dag_status:2, start_time:1727337655676645, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678608] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=1] schedule one task(task=0x7f92728392e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.678621] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.678624] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:00:55.678626] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:00:55.678629] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:00:55.678632] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:2, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337655632142, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:00:55.678658] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101063, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b0060, count:8, allocator:0x7f92636927d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b03e0, local_datums:0x7f92684b03e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:55.678673] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=40] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:00:55.678678] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241548806527000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.678707] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=38][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.678743] ERROR issue_dba_error (ob_log.cpp:1875) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.678746] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.678755] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=0] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337655632228, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.678767] ERROR issue_dba_error (ob_log.cpp:1875) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.678771] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.678774] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337655678619, end_time=1727337655678774, runtime=155, *this={this:0x7f92728392e0, type:1, status:2, dag:{ObIDag:{this:0x7f9272836b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F5-0-0, dag_ret:0, dag_status:2, start_time:1727337655676645, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678783] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2138, dag_cnt=5, dag_cnts_[dag.get_type()]=5, &dag=0x7f9272836b00, dag={ObIDag:{this:0x7f9272836b00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F5-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655676645, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.678780] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=0] partition merge iter row count(i=0, row_count=141, ghost_row_count=0, pkey={tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337655632275, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.678791] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F5-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655676640, task_id:YB42C0A8002D-0006230056D8C6F5-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.678801] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655678512, end_time=1727337655678801, runtime=289, *this={this:0x7f9272838f50, type:1, status:2, dag:{ObIDag:{this:0x7f9272837a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F8-0-0, dag_ret:0, dag_status:2, start_time:1727337655677694, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678810] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1116, dag_cnt=4, dag_cnts_[dag.get_type()]=4, &dag=0x7f9272837a90, dag={ObIDag:{this:0x7f9272837a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F8-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655677694, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.678818] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F8-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655677689, task_id:YB42C0A8002D-0006230056D8C6F8-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.678867] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337655632345, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.678877] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.678896] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.678899] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.678904] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101067}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.678910] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=329, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ec060, count:6, allocator:0x7f928642a7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f92664ec390, local_datums:0x7f92664ec390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:00:55.678921] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:101067}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101067}, table_count:1, [{i:0, table_key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b86a2d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655678366, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=535us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.678981] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272838a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FB-0-0, dag_ret:0, dag_status:2, start_time:1727337655678352, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d4a0, output_block_cnt_arr:0x7f9264b1d4a8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337655678366, estimated_finish_time:1727337672678981}) [2024-09-26 16:00:55.678989] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9272838de0, type:15, status:2, dag:{ObIDag:{this:0x7f9272838a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FB-0-0, dag_ret:0, dag_status:2, start_time:1727337655678352, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.678994] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=4] task finish process(ret=0, start_time=1727337655678365, end_time=1727337655678994, runtime=629, *this={this:0x7f9272838de0, type:15, status:2, dag:{ObIDag:{this:0x7f9272838a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FB-0-0, dag_ret:0, dag_status:2, start_time:1727337655678352, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.679017] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=3] schedule one task(task=0x7f9264b36030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.679038] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752511928, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.679055] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=13] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.679056] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.679059] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.679066] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:121}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752511928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.679097] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.679100] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.679101] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=29] succeed to build merge ctx(tablet_id={id:121}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655678616, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=448us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.679111] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=0] partition merge iter row count(i=0, row_count=47, ghost_row_count=0, pkey={tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337655632291, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.679125] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9272834be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EF-0-0, dag_ret:0, dag_status:2, start_time:1727337655674518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f9268515480, output_block_cnt_arr:0x7f9268515488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337655678616, estimated_finish_time:1727337672679124}) [2024-09-26 16:00:55.679130] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655678883, end_time=1727337655679130, runtime=247, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9272837fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F9-0-0, dag_ret:0, dag_status:2, start_time:1727337655677965, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.679132] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9272834fa0, type:15, status:2, dag:{ObIDag:{this:0x7f9272834be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EF-0-0, dag_ret:0, dag_status:2, start_time:1727337655674518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:55.679138] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=5] task finish process(ret=0, start_time=1727337655678615, end_time=1727337655679137, runtime=522, *this={this:0x7f9272834fa0, type:15, status:2, dag:{ObIDag:{this:0x7f9272834be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EF-0-0, dag_ret:0, dag_status:2, start_time:1727337655674518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:55.679139] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1174, dag_cnt=3, dag_cnts_[dag.get_type()]=3, &dag=0x7f9272837fc0, dag={ObIDag:{this:0x7f9272837fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6F9-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655677965, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.679146] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6F9-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655677960, task_id:YB42C0A8002D-0006230056D8C6F9-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.679175] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337655632306, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:00:55.679183] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=25, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:00:55.679196] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:00:55.679199] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:00:55.679205] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:332}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:27, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:00:55.679223] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:332}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:332}, table_count:1, [{i:0, table_key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8648b00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337655678365, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=838us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:00:55.679245] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f92728384f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FA-0-0, dag_ret:0, dag_status:2, start_time:1727337655678218, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db7490, output_block_cnt_arr:0x7f9287db7498, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337655678365, estimated_finish_time:1727337672679245}) [2024-09-26 16:00:55.679262] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=16] succeed to init merge ctx(task={this:0x7f92728388b0, type:15, status:2, dag:{ObIDag:{this:0x7f92728384f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FA-0-0, dag_ret:0, dag_status:2, start_time:1727337655678218, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.679267] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=5] task finish process(ret=0, start_time=1727337655678363, end_time=1727337655679267, runtime=904, *this={this:0x7f92728388b0, type:15, status:2, dag:{ObIDag:{this:0x7f92728384f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FA-0-0, dag_ret:0, dag_status:2, start_time:1727337655678218, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.679285] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=3] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.679319] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101067, range=0x7f92aaad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a36060, count:8, allocator:0x7f92c1d9c7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263a363e0, local_datums:0x7f9263a363e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:00:55.679550] ERROR issue_dba_error (ob_log.cpp:1875) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.679555] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.679563] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=0] partition merge iter row count(i=0, row_count=227, ghost_row_count=0, pkey={tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337655632345, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.679581] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337655679286, end_time=1727337655679581, runtime=295, *this={this:0x7f9264b36030, type:1, status:2, dag:{ObIDag:{this:0x7f9272838a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FB-0-0, dag_ret:0, dag_status:2, start_time:1727337655678352, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.679599] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1247, dag_cnt=2, dag_cnts_[dag.get_type()]=2, &dag=0x7f9272838a20, dag={ObIDag:{this:0x7f9272838a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FB-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655678352, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.679607] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FB-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337655678348, task_id:YB42C0A8002D-0006230056D8C6FB-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.679656] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=332, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:25, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:27}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a36060, count:6, allocator:0x7f92865b67d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:27, datum_buffer:{capacity:32, datums:0x7f9263a36390, local_datums:0x7f9263a36390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,]}) [2024-09-26 16:00:55.679711] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.679714] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.679721] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=1] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337655632306, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.679736] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655679633, end_time=1727337655679736, runtime=103, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f92728384f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FA-0-0, dag_ret:0, dag_status:2, start_time:1727337655678218, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:00:55.679743] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1525, dag_cnt=1, dag_cnts_[dag.get_type()]=1, &dag=0x7f92728384f0, dag={ObIDag:{this:0x7f92728384f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FA-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655678218, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:00:55.679749] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FA-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655678213, task_id:YB42C0A8002D-0006230056D8C6FA-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.682046] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=3] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.682116] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=121, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a36060, count:5, allocator:0x7f926367a7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a36360, local_datums:0x7f9263a36360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:00:55.682134] INFO [STORAGE] dump_info (ob_sstable_struct.cpp:262) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=15] dump merge info(msg="macro block builder close", output_row_per_s=0, new_macro_KB_per_s=0, *this={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:39, merge_start_time:1727337655682095, merge_finish_time:1727337655682134, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:0, max:0, sum:0, count:0}}, {type:"cost_time", info:{min:0, max:0, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:0, max:0, sum:0, count:0}}, {type:"incremental_row_count", info:{min:0, max:0, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:0, snapshot_version:0, start_scn:0, end_scn:0}, macro_id_list:"", comment:""}) [2024-09-26 16:00:55.682149] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=14] partition merge iter row count(i=0, row_count=0, ghost_row_count=0, pkey={tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, table={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337655632142, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:00:55.682160] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1544) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=9] merge macro blocks ok(idx_=0, task={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272834be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EF-0-0, dag_ret:0, dag_status:2, start_time:1727337655674518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:55.682170] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=6] task finish process(ret=0, start_time=1727337655682092, end_time=1727337655682170, runtime=78, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9272834be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EF-0-0, dag_ret:0, dag_status:2, start_time:1727337655674518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:55.682208] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=4] schedule one task(task=0x7f926652a250, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:00:55.682223] INFO [STORAGE.COMPACTION] get_merged_sstable (ob_tablet_merge_task.cpp:1285) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=5] create new merged sstable(ctx.param_.tablet_id_={id:121}, snapshot_version=1726241132764173999, ctx.param_.merge_type_=3, ctx.create_snapshot_version_=0, table_mode_flag=0) [2024-09-26 16:00:55.682230] INFO [STORAGE] close (ob_index_block_builder.cpp:815) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=3] succeed to close sstable index builder(res={root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_block_ids_.count():0, other_block_ids_.count():0, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, data_column_cnt:10, data_column_checksums:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], row_count:0, max_merged_trans_version:0, contain_uncommitted_row:false, occupy_size:0, original_size:0, data_checksum:0, use_old_macro_block_count:0, compressor_type:1, root_row_store_type:0, nested_offset:0, nested_size:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}) [2024-09-26 16:00:55.682246] INFO [STORAGE] init (ob_sstable.cpp:118) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=9] succeeded to init sstable(ret=0, this={ObITable:{this:0x7f9268514128, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9268514128, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:null, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:55.682264] INFO [STORAGE.COMPACTION] create_sstable (ob_tablet_merge_ctx.cpp:367) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=17] succeed to merge sstable(param={table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_logic_seq:0, schema_version:1710828752511928, create_snapshot_version:0, progressive_merge_round:0, progressive_merge_step:0, is_ready_for_read:true, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, index_type:0, root_block_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), root_block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, root_row_store_type:0, latest_row_store_type:1, data_index_tree_height:0, data_block_macro_meta_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), data_block_macro_meta:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, use_old_macro_block_count:0, row_count:0, rowkey_column_cnt:5, column_cnt:10, column_checksums:[], data_checksum:0, occupy_size:0, original_size:0, max_merged_trans_version:1726241132764173999, ddl_scn:{val:0, v:0}, contain_uncommitted_row:false, is_meta_root:false, compressor_type:1, encrypt_id:0, master_key_id:0, recycle_version:0, nested_offset:0, nested_size:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}, table_key={tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_merge_info={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:3518, merge_start_time:1727337655678616, merge_finish_time:1727337655682134, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"cost_time", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"incremental_row_count", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:1, snapshot_version:0, start_scn:1726211867305630000, end_scn:1726241548806527000}, macro_id_list:"", comment:""}) [2024-09-26 16:00:55.682515] INFO [STORAGE] build_new_table_store (ob_tablet_table_store.cpp:1206) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=19] succeed to build new table store(major_tables_={cnt:1, is_inited:true}, minor_tables_={cnt:3, is_inited:true}, memtables_={count:1}, PRINT_TS(*this)={ObTabletTableStore_Pretty:this:0x7f92a9855ca8, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, memtables:{count:1}, is_ready_for_read:true, table_array:{ [2888] [ YB42C0A8002D-0006230056D8C6EF-0-0 ] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [2888] [ YB42C0A8002D-0006230056D8C6EF-0-0 ] MAJOR MAJOR 1725120003420849000 1725120003420849000 {val:0, v:0} {val:1726077600415496000, v:0} 0 false [2888] [ YB42C0A8002D-0006230056D8C6EF-0-0 ] MINOR MINOR 1726178585269925002 1726178585269925002 {val:1726077259159067000, v:0} {val:1726178585269925002, v:0} 0 false [2888] [ YB42C0A8002D-0006230056D8C6EF-0-0 ] MINI 1726211867106157000 1726211867106157000 {val:1726178585269925002, v:0} {val:1726211867305630000, v:0} 0 false [2888] [ YB42C0A8002D-0006230056D8C6EF-0-0 ] MINI 1726241132764173999 1726241132764173999 {val:1726211867305630000, v:0} {val:1726241548806527000, v:0} 0 false [2888] [ YB42C0A8002D-0006230056D8C6EF-0-0 ] MEM ACTIVE 9223372036854775807 18446744073709547609 {val:1726241548806527000, v:0} {val:4611686018427387903, v:0} 3 unused }}) [2024-09-26 16:00:55.682735] INFO [MDS] init_for_evict_medium_info (ob_tablet_mds_data.cpp:414) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=16] succeeded to init mds data(ret=0, finish_medium_scn=1726077600415496000, src_medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, src_extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}) [2024-09-26 16:00:55.682753] INFO [STORAGE] init_for_merge (ob_tablet.cpp:363) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=8] succeeded to init tablet for mini/minor/major merge(ret=0, param={sstable:0x7f9268514128, snapshot_version:1726241132764173999, clog_checkpoint_scn:{val:1726241548806527000, v:0}, multi_version_start:1726178585269925002, need_report:false, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, rebuild_seq:0, update_with_major_flag:false, need_check_sstable:false, ddl_info:{keep_old_ddl_sstable:true, ddl_start_scn:{val:0, v:0}, ddl_commit_scn:{val:0, v:0}, ddl_checkpoint_scn:{val:0, v:0}, ddl_snapshot_version:0, ddl_execution_id:-1, data_format_version:0}, allow_duplicate_sstable:false, merge_type:"MINI_MERGE", need_check_transfer_seq:true, transfer_seq:0}, old_tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337655631503000, hold_ref_cnt:true, ref_cnt:3, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337655632142, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}, this={this:0x7f92728f6060, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x7f92a9856450, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726241548806527000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726241132764173999, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92728f6670, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}, {i:2, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726241132764173999, upper_trans_version:1726241132764173999}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x7f92728f6520, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f92728f7880, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f92728f7950, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{redefined:false, snapshot_version:0, schema_version:0, data_tablet_id:{id:0}, hidden_tablet_id:{id:0}, lob_meta_tablet_id:{id:50121}, lob_piece_tablet_id:{id:60121}, is_old_mds:true}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:00:55.682844] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=80] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9263a4a060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}}, ref_cnt:0, upper_trans_version:1725120003420849000, timestamp:0}, this:0x7f9263a4a060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1725120003420849000, max_merged_trans_version:1725120003420849000, data_macro_block_count:1, nested_size:4096, nested_offset:589824, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:84, row_count_:14, occupy_size_:1604, original_size_:1914, data_checksum_:3587602083, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:1, data_micro_block_count_:1, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:2, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1725120003420849000, max_merged_trans_version_:1725120003420849000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9263a4a310, column_checksum_count:10, data_root_info:{addr:[0-0-0-3458764933653594112-0](offset=0,size=391,type=3,seq=0), block_data:{buf:0x7f9263a4a360, size:391, extra_buf:0x7f9263a4a4e7, extra_size:424, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-3458765009889263616-0](offset=0,size=462,type=3,seq=0), block_data:{buf:0x7f9263a4a68f, size:462, extra_buf:null, extra_size:0, type:1}}, data_block_count:1, other_block_count:0, linked_block_count:0, data_block_ids:0x7f9263a4a85d, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:true, nested_offset:589824, nested_size:4096}}, valid_for_reading:true}) [2024-09-26 16:00:55.682865] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=19] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9263a4a060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}}, ref_cnt:0, upper_trans_version:1726178585269925002, timestamp:0}, this:0x7f9263a4a060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726178585269925002, max_merged_trans_version:1726178585269925002, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726178585269925002, v:0}, meta:{basic_meta:{version:1, length:94, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726178585269925002, max_merged_trans_version_:1726178585269925002, recycle_version:1726077600415496000, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726178585269925002, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:1, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9263a4a310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:55.682876] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=10] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9263a4a060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}}, ref_cnt:0, upper_trans_version:1726211867106157000, timestamp:0}, this:0x7f9263a4a060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726211867106157000, max_merged_trans_version:1726211867106157000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726211867305630000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726211867106157000, max_merged_trans_version_:1726211867106157000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726211867305630000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9263a4a310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:55.682887] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=10] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9263a4a060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9263a4a060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9263a4a310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:00:55.682897] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=9][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:55.682934] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:00:55.682936] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf9de5b1 0xf9dcfd6 0xe51c6fb 0xe519c03 0xe518f89 0xe516f8c 0xf01158d 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:00:55.682955] INFO [STORAGE] reset (ob_tablet.cpp:159) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=0] reset tablet(this=0x7f92728f6060, ls_id={id:1}, tablet_id={id:121}, lbt()="0x121f1f9c 0xe52cc09 0xe52c6c5 0xe52bd13 0x4fd348f 0xf01110c 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:55.682961] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1250) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=0][errcode=-4184] sstable merge finish(ret=-4184, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:true, merge_dag:{ObIDag:{this:0x7f9272834be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EF-0-0, dag_ret:0, dag_status:2, start_time:1727337655674518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f9268515480, output_block_cnt_arr:0x7f9268515488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337655678616, estimated_finish_time:1727337672679124}, compaction_filter:NULL, time_guard:total=3.67ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, task={this:0x7f926652a250, type:16, status:2, dag:{ObIDag:{this:0x7f9272834be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EF-0-0, dag_ret:0, dag_status:2, start_time:1727337655674518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:55.682988] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337655682221, end_time=1727337655682987, runtime=766, *this={this:0x7f926652a250, type:16, status:2, dag:{ObIDag:{this:0x7f9272834be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EF-0-0, dag_ret:0, dag_status:2, start_time:1727337655674518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:00:55.682996] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=8477, dag_cnt=0, dag_cnts_[dag.get_type()]=0, &dag=0x7f9272834be0, dag={ObIDag:{this:0x7f9272834be0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6EF-0-0, dag_ret:-4184, dag_status:5, start_time:1727337655674518, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}) [2024-09-26 16:00:55.683002] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6EF-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337655674514, task_id:YB42C0A8002D-0006230056D8C6EF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:00:55.703763] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337655336910]}) [2024-09-26 16:00:55.706697] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.706703] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.712657] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:55.712772] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=63, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:55.713382] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=314570, clean_num=31457) [2024-09-26 16:00:55.736978] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.736999] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.744790] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:55.767069] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:55.767077] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.769184] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:55.769199] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337655769181) [2024-09-26 16:00:55.769204] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337655669007, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:55.769219] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223385004946, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.769225] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:55.769232] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:55.769237] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:55.769239] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:55.769245] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.769247] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.769249] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337655769207) [2024-09-26 16:00:55.782228] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:55.794599] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [2925][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=15] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f92b77d02b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-09-26 16:00:55.797527] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.797534] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.804326] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337655336910]}) [2024-09-26 16:00:55.818070] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=22][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337655336910]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:55.818080] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=10][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:55.820940] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.821018] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.821030] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.821037] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.821773] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=25] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.821793] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.821804] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.821809] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.821813] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.821816] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=26] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.821823] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:55.827767] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:55.827774] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.837619] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:55.837628] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337655837616], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:55.837635] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337655837616], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:55.837638] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:55.837643] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:55.858074] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.858096] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.869793] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:55.869804] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337655869791) [2024-09-26 16:00:55.869809] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337655769207, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:55.869824] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223385106259, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.869830] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:55.869838] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:55.869844] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:55.869846] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:55.869852] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.869855] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.869858] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337655869812) [2024-09-26 16:00:55.873730] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337656803061, now=[mts=1727337654873263]) [2024-09-26 16:00:55.873737] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337656803061, now=[mts=1727337654873263], now0=[mts=1727337654873263], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:55.873742] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a3127820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500811, assoc_session_id:3221500811, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337654872555, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:55.873758] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22f01f8, id:3221500811, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3127820}) [2024-09-26 16:00:55.873766] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500811, *tx_desc={this:0x7f92a3127820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500811, assoc_session_id:3221500811, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337654872555, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:55.873773] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=3221500811, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3127820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500811, assoc_session_id:3221500811, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337654872555, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337654873242, use_das=false, nested_level=0, session={this:0x7f92a22f01f8, id:3221500811, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3127820}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:55.873789] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:55.873793] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:55.873796] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:55.873798] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:55.873801] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:55.873804] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:55.873809] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=999252, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337656873061) [2024-09-26 16:00:55.873812] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:55.873819] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:55.873821] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=1][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:55.873822] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:55.873827] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:55.873849] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2169) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2] need check schema(new_schema_version=1, cached_tenant_schema_version=1, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = ? ORDER BY row_id, column_name) [2024-09-26 16:00:55.873868] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:55.873871] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:55.874706] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1727337655874705, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007024, add_bytes: 0 [2024-09-26 16:00:55.888299] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:55.888306] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.890772] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=3] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-09-26 16:00:55.890783] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=7] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=691) [2024-09-26 16:00:55.904470] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337655837616]}) [2024-09-26 16:00:55.913849] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:55.913950] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=49, refresh_score_time=13, wash_time=2) [2024-09-26 16:00:55.914570] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=346027, clean_num=31457) [2024-09-26 16:00:55.918605] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:55.918612] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.948894] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:55.948911] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.970429] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:55.970441] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337655970426) [2024-09-26 16:00:55.970447] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337655869812, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:55.970463] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223385206915, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:55.970469] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:55.970477] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:55.970482] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:55.970496] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:55.970503] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.970507] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:55.970510] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337655970450) [2024-09-26 16:00:55.979220] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:55.979226] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:55.982897] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:55.985026] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:56.004592] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337655837616]}) [2024-09-26 16:00:56.009442] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:56.009450] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.024596] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:56.024608] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-09-26 16:00:56.024613] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:56.024616] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-09-26 16:00:56.024620] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:56.024624] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:56.024628] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-09-26 16:00:56.024630] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:56.024632] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:56.024634] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:56.024637] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] resolve table failed(ret=-5019) [2024-09-26 16:00:56.024640] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:2822) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=3][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-09-26 16:00:56.024643] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2676) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] resolve joined table failed(ret=-5019) [2024-09-26 16:00:56.024644] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:56.024647] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:56.024652] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:56.024655] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:56.024661] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:56.024663] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:56.024675] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:56.024677] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:56.024680] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-09-26 16:00:56.024682] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:56.024687] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:56.024693] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:56.024699] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:56.024702] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:56.024703] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:56.024710] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:56.024713] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA4-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:56.024718] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [3024][BlackListServic][T0][YB42C0A8002D-0006230056284FA4-0-0] [lt=4][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-09-26 16:00:56.024721] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:56.024724] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:56.024726] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:56.024728] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337656024495, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:56.024732] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:56.024735] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:222) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:00:56.024738] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:398) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2] start to print blacklist info [2024-09-26 16:00:56.024762] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2] ls blacklist refresh finish(cost_time=523) [2024-09-26 16:00:56.026212] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185928-0-0] [lt=33][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:56.026220] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185928-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:56.026229] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185928-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:56.026234] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185928-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:56.031113] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] new mysql sessid created(conn.sessid_=3221500817, support_ssl=false) [2024-09-26 16:00:56.031140] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221500817, sess.client_addr_="127.0.0.1:36372") [2024-09-26 16:00:56.031147] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] accept one succ(*s={this:0x7f92be35cf90, session_id:3221500817, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:56.031171] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:56.031179] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:56.031215] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA4-0-0] [lt=8] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:56.031412] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA4-0-0] [lt=9][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:56.031419] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA4-0-0] [lt=6][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"XuezTtj:"9:U~=Tk?:Ax"}, ret=-4043) [2024-09-26 16:00:56.031426] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA4-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:56.031429] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA4-0-0] [lt=2][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:56.031437] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA4-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:56.031457] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA4-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500817, proxy_sessid:0}) [2024-09-26 16:00:56.031461] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA4-0-0] [lt=4] mark session id unused(sessid=3221500817) [2024-09-26 16:00:56.031510] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA4-0-0] [lt=46][errcode=0] server close connection(sessid=3221500817, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:56.031514] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA4-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500817, proxy_sessid=0) [2024-09-26 16:00:56.031516] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA4-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:56.031522] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA4-0-0] [lt=5] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500817, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:56.031540] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f92be35cf90, session_id:3221500817, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337656031121, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:56.031554] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221500817, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:56.031558] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] can close safely, do destroy(*s={this:0x7f92be35cf90, session_id:3221500817, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337656031121, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:56.031563] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500817, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:56.040739] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:56.040746] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.057640] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:56.057653] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:56.057657] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:56.057676] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:56.057681] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:56.057684] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:56.057687] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:56.057690] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:56.057692] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:56.057694] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:56.057696] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:56.057699] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:56.057701] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:56.057706] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:56.057713] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:56.057715] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:56.057718] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:56.057721] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:56.057725] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:56.057729] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:56.057731] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:56.057738] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=6][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:56.057744] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:56.057747] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:56.057749] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:56.057755] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:56.057758] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:56.057762] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85891-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:56.057765] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:56.057768] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85891-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:56.057772] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85891-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:56.057775] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337656057572, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:56.057778] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85891-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:56.057781] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85891-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:56.057804] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85891-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:56.057807] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85891-0-0] [lt=3][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:56.066692] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=8] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:56.067036] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:56.067044] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:56.067048] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:56.067051] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:56.067054] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:56.067056] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:56.067059] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:56.067062] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:56.067063] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:56.067065] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:56.067066] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:56.067069] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:56.067070] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:56.067072] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:56.067077] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:56.067079] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:56.067081] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:56.067083] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:56.067085] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:56.067087] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:56.067089] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:56.067094] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:56.067098] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:56.067100] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:56.067101] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:56.067107] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:56.067110] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:56.067112] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D7A-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:56.067115] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:56.067117] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:56.067119] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:56.067121] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337656066969, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:56.067124] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:56.067163] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:56.070889] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:56.070899] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:56.070906] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337656070887) [2024-09-26 16:00:56.070933] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:56.070936] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337655970450, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:56.070944] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:56.070941] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.070953] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:56.070956] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:56.071227] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:56.071238] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:56.071241] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:56.071244] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:56.071261] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=16][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:56.071264] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:56.071266] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:56.071271] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=5][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:56.071273] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:56.071275] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:56.071276] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:56.071278] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:56.071280] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:56.071281] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:56.071286] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:56.071288] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:56.071290] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:56.071293] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:56.071297] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:56.071300] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:56.071302] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:56.071306] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:56.071311] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:56.071315] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:56.071316] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:56.071321] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:56.071324] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:56.071326] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576E-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:56.071329] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:56.071331] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:56.071333] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:56.071336] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337656071179, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:56.071339] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:56.071341] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:56.071362] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:56.071366] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337656071365, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=416, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:56.071373] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:56.071377] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:56.071393] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223385307033, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:56.071398] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:56.071404] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:56.071409] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:56.071412] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:56.071417] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.071419] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.071421] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337656071381) [2024-09-26 16:00:56.087564] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:56.101253] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:56.101262] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.101274] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:56.106153] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337655837616]}) [2024-09-26 16:00:56.115049] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:56.115151] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=61, refresh_score_time=16, wash_time=1) [2024-09-26 16:00:56.115754] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=377484, clean_num=31457) [2024-09-26 16:00:56.126679] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: SHARED_BLOCK_ReaderWriter_IO, mode: read, size: 4096.00, iops: 52.68, rt: 2217.28) [2024-09-26 16:00:56.126689] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:56.126721] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 9322.00) [2024-09-26 16:00:56.126728] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:56.126735] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:56.126738] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:56.131548] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:56.131553] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.162002] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:56.162010] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.168686] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=19][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337655167580]) [2024-09-26 16:00:56.168693] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337655167580], now0=[mts=1727337655167580], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:56.168700] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337655167520, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:56.168715] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:56.168722] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337655167520, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:56.168729] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337655167520, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:56.168757] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=27][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:56.168762] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:56.168765] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:56.168767] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:56.168769] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:56.168773] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=18, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:56.168777] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=18000, remain_us=10825556, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:56.171594] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:56.171602] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337656171590) [2024-09-26 16:00:56.171607] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337656070941, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:56.171622] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223385408050, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:56.171628] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:56.171635] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:56.171640] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:56.171642] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:56.171648] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.171651] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.171652] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337656171610) [2024-09-26 16:00:56.183310] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:56.187247] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:18, local_retry_times:18, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:56.187260] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:56.187263] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:56.187265] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:56.187270] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:56.187306] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:56.187308] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:56.192092] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:56.192099] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.200846] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=2] ====== check clog disk timer task ====== [2024-09-26 16:00:56.200854] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=6] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:56.200864] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:56.206751] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337655837616]}) [2024-09-26 16:00:56.206767] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D95A-0-0] [lt=11][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337656206745}) [2024-09-26 16:00:56.222214] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=28] success to notify_need_writing_throttling True [2024-09-26 16:00:56.222225] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.252023] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:56.252034] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:56.252039] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:56.252042] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:56.252046] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:56.252050] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=4][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:56.252054] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:56.252056] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:56.252059] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:56.252061] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:56.252062] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:56.252065] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:56.252067] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:56.252069] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:56.252075] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:56.252077] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:56.252082] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:56.252084] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:56.252086] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:56.252089] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:56.252092] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:56.252097] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:56.252103] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:56.252105] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:56.252106] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:56.252113] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:56.252117] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:56.252120] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:56.252122] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:56.252125] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:56.252127] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:56.252129] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337656251952, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:56.252133] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:56.252135] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:56.252163] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=3][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:56.252166] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=3][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:56.252169] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:56.252171] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=1][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:56.252173] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=1][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:56.252177] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=3][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:56.252179] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A0-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:56.252662] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:56.252671] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.272224] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:56.272253] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337656272222) [2024-09-26 16:00:56.272259] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337656171610, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:56.272277] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223385507696, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:56.272283] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:56.272292] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:56.272297] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:56.272303] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:56.272310] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.272313] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.272315] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337656272263) [2024-09-26 16:00:56.282952] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:56.282958] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.307337] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337655837616]}) [2024-09-26 16:00:56.313195] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=50][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337656242330, now=[mts=1727337655312235]) [2024-09-26 16:00:56.313214] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=17][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337656242330, now=[mts=1727337655312235], now0=[mts=1727337655312235], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:56.313220] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269164fe0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337655312079, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:56.313225] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:56.313237] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269164fe0}) [2024-09-26 16:00:56.313241] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.313245] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269164fe0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337655312079, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:56.313252] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269164fe0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337655312079, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337654312572, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269164fe0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:56.313269] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:56.313274] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:56.313280] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=5][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:56.313283] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:56.313285] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:56.313288] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:56.313295] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=6][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:56.313302] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:56.313314] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=12][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:56.313316] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:56.313320] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C82-0-0] [lt=1][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:56.313324] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=3] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000752) [2024-09-26 16:00:56.313327] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:56.313330] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:56.313333] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:56.313336] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:56.313338] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:56.313344] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:56.313370] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:56.313375] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:56.313377] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:56.313380] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:56.313384] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:56.313387] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:56.313494] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C82-0-0] [lt=4][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:56.313503] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:56.313505] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:56.313523] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=1] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:56.313532] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=8][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:56.313810] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:00:56.313818] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=7][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:56.316282] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:56.316383] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:56.316996] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=408941, clean_num=31457) [2024-09-26 16:00:56.338712] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.338720] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337656338709], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:56.338727] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337656338709], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:56.338730] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=2] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=5153, get_gts_with_stc_cnt=11985, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-09-26 16:00:56.338734] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337656338709]}) [2024-09-26 16:00:56.343477] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:56.343485] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.372287] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:56.372299] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337656372284) [2024-09-26 16:00:56.372304] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337656272263, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:56.372320] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223385608318, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:56.372327] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:56.372334] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:56.372339] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:56.372342] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:56.372348] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.372350] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.372352] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337656372307) [2024-09-26 16:00:56.373751] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:56.373756] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.383537] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:56.390364] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=9] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:56.390398] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=3] current monitor number(seq_=-1) [2024-09-26 16:00:56.404055] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:56.404062] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.407952] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337656338709]}) [2024-09-26 16:00:56.411860] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D4097-0-0] [lt=11][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:56.434353] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:56.434362] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.436284] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337656436283, bytes: 14690896, bw: 0.003375 MB/s, add_ts: 1000160, add_bytes: 3540 [2024-09-26 16:00:56.464588] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.464605] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.464612] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.464613] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=24][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.464617] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.464618] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.464625] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:00:56.464635] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.464630] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.464637] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.464639] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.465563] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.465572] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.465577] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.465592] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.465595] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.465597] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.465601] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.465602] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.465604] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.466525] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.466535] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.466540] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.466544] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.466555] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.466558] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.466560] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.466561] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.466563] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.467524] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.467527] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.467529] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.467532] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.467534] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.467535] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.467535] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.467538] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.467543] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.468507] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.468512] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.468514] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.468514] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.468518] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.468517] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.468519] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.468520] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.468521] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.469481] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.469482] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.469485] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:56.469486] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.469483] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.470458] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.470461] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.470477] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.471459] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.471475] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.471474] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.472421] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:56.472431] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337656472418) [2024-09-26 16:00:56.472436] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337656372307, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:56.472452] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223385707956, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:56.472457] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:56.472464] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:56.472492] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:56.472495] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:56.472501] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.472502] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.472504] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337656472438) [2024-09-26 16:00:56.472515] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.472523] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.472527] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.474387] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.474391] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.474394] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.475361] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.475369] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.475368] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.476326] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.476379] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.476421] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.477261] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.477280] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.477291] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.478282] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.478286] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.478291] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.479255] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.479272] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.479255] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.480247] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.480249] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.480249] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.481194] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.481272] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.481283] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.482164] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.482164] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.482217] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.483139] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.483151] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.483160] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.484111] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.485119] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.485131] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.485139] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.486094] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.486100] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.486099] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.487061] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.487062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.487052] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.488030] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.488034] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.488037] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.489019] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.489029] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.489041] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.489993] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.489996] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.490001] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.490951] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.490957] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.490967] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.491938] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.491953] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.491953] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.492945] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.492958] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.492959] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.493877] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.493885] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.493885] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.494874] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.494942] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.494949] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.494985] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:56.494991] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.495841] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.495844] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.495846] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.496838] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.496846] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.496846] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.497809] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.497810] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.497816] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:56.508458] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337656338709]}) [2024-09-26 16:00:56.517449] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:56.517644] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=17] Wash time detail, (compute_wash_size_time=82, refresh_score_time=28, wash_time=2) [2024-09-26 16:00:56.518524] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=440398, clean_num=31457) [2024-09-26 16:00:56.522019] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185929-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:56.522035] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185929-0-0] [lt=15][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:56.522048] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185929-0-0] [lt=12][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:56.522054] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185929-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:56.525054] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:56.525064] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.546647] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6578, mem_used=12478464) [2024-09-26 16:00:56.555428] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:00:56.555439] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.565196] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=9] gc stale ls task succ [2024-09-26 16:00:56.573021] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:56.573033] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337656573019) [2024-09-26 16:00:56.573039] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337656472438, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:56.573043] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:56.573054] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:56.573057] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:56.573425] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576F-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:56.573439] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576F-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:56.573443] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576F-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:56.573447] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576F-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:56.573450] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576F-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:56.573452] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576F-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:56.573459] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576F-0-0] [lt=5][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:56.573476] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576F-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:56.573499] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788576F-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:56.573534] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337656573533, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=484, wlock_time=10, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:56.573543] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:56.573559] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223385809485, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:56.573565] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:56.573572] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:56.573577] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:56.573580] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:56.573586] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.573588] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.573590] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337656573548) [2024-09-26 16:00:56.583766] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:56.585700] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=28] success to notify_need_writing_throttling True [2024-09-26 16:00:56.585708] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.601363] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.602319] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.602324] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.602362] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.609136] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=38] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337656338709]}) [2024-09-26 16:00:56.616053] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:56.616061] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.646282] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:56.646290] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.663872] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=5] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:56.671686] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:56.674569] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:56.674578] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337656674567) [2024-09-26 16:00:56.674592] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337656573041, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:56.674610] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223385911036, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:56.674617] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:56.674624] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:56.674629] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:56.674632] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:56.674638] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.674641] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.674643] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337656674597) [2024-09-26 16:00:56.676547] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:56.676555] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.706894] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:56.706902] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.709762] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337656338709]}) [2024-09-26 16:00:56.717612] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4184, dropped:518, tid:2871}]) [2024-09-26 16:00:56.718695] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:56.718813] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=52, refresh_score_time=20, wash_time=2) [2024-09-26 16:00:56.719422] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=471855, clean_num=31457) [2024-09-26 16:00:56.737187] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:56.737192] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.750838] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:56.767399] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:56.767408] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.775223] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:56.775239] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337656775220) [2024-09-26 16:00:56.775245] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337656674597, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:56.775272] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=223386011655, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:56.775278] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:56.775287] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:56.775292] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:56.775294] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:56.775300] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.775302] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.775304] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337656775259) [2024-09-26 16:00:56.784024] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:56.788878] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=29][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337656338709]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:00:56.788902] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=23][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:00:56.797651] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:56.797663] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.810352] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337656338709]}) [2024-09-26 16:00:56.820256] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [2915][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=10] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-09-26 16:00:56.821115] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.821142] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.821123] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.821150] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.821156] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.821161] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.821167] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.821220] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.822086] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.822098] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.822105] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:56.827925] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:56.827933] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.840738] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:56.840749] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337656840735], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:56.840757] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337656840735], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:56.840761] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:56.840764] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:56.858200] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:56.858226] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.872950] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337656803061, now=[mts=1727337655873867]) [2024-09-26 16:00:56.872968] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=18][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337656803061, now=[mts=1727337655873867], now0=[mts=1727337655873867], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:56.872977] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a3127820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500811, assoc_session_id:3221500811, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337655872805, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:56.872999] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=20][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22f01f8, id:3221500811, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3127820}) [2024-09-26 16:00:56.873008] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500811, *tx_desc={this:0x7f92a3127820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500811, assoc_session_id:3221500811, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337655872805, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:56.873022] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=3221500811, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3127820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500811, assoc_session_id:3221500811, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337655872805, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337654873242, use_das=false, nested_level=0, session={this:0x7f92a22f01f8, id:3221500811, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3127820}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:56.873042] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=19][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:56.873047] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:56.873050] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:56.873053] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:56.873056] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:56.873061] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:56.873066] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4] already timeout, do not need sleep(sleep_us=-5, remain_us=-5, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, timeout_timestamp=1727337656873061) [2024-09-26 16:00:56.873070] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:56.873078] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:56.873080] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:56.873082] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:56.873087] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:56.873127] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1764) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4012] get gts timeout(ret=-4012, expire_ts=1727337656803061, current_time=1727337656872930) [2024-09-26 16:00:56.873133] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5][errcode=-4012] acquire global snapshot fail(ret=-4012, gts_ahead=0, expire_ts=1727337656803061, now=[mts=1727337656873127], now0=[mts=1727337656873127], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:56.873136] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4012] acquire global snapshot fail(ret=-4012, tx={this:0x7f92a3127820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500811, assoc_session_id:3221500811, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337656872930, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:56.873142] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=6][errcode=-4012] fail to get snapshot(ret=-4012, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22f01f8, id:3221500811, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3127820}) [2024-09-26 16:00:56.873145] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4012] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4012, session_id=3221500811, *tx_desc={this:0x7f92a3127820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500811, assoc_session_id:3221500811, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337656872930, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:56.873150] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=5] start stmt(ret=-4012, auto_commit=true, session_id=3221500811, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a3127820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500811, assoc_session_id:3221500811, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337656872930, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337654873242, use_das=false, nested_level=0, session={this:0x7f92a22f01f8, id:3221500811, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a3127820}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:56.873160] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=10][errcode=-4012] fail to start stmt(ret=-4012, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:56.873162] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4012] fail start stmt(ret=-4012) [2024-09-26 16:00:56.873164] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4012] execute plan failed(ret=-4012) [2024-09-26 16:00:56.873165] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=1][errcode=-4012] open result set failed(ret=-4012) [2024-09-26 16:00:56.873167] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:56.873171] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=4][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:56.873174] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:56.873177] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:56.873178] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=1][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:00:56.873180] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-09-26 16:00:56.873182] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=1][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:56.873184] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=1999942) [2024-09-26 16:00:56.873187] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885089-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:56.873191] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:56.873194] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:56.873197] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:56.873202] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8620, start=1727337654873239, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:56.873205] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:00:56.873208] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:56.873238] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:56.873242] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:56.873245] WDIAG [SHARE] get_snapshot_gc_scn (ob_global_stat_proxy.cpp:164) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:56.873249] WDIAG [STORAGE] get_global_info (ob_tenant_freeze_info_mgr.cpp:811) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] fail to get global info(ret=-4012, tenant_id=1) [2024-09-26 16:00:56.873252] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:954) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] failed to get global info(ret=-4012) [2024-09-26 16:00:56.873254] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1008) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to try update info(tmp_ret=-4012, tmp_ret="OB_TIMEOUT") [2024-09-26 16:00:56.874805] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337656874804, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000099, add_bytes: 0 [2024-09-26 16:00:56.875796] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:56.875818] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337656875793) [2024-09-26 16:00:56.875823] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337656775259, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:56.875843] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223386111297, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:56.875848] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:56.875856] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:56.875861] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:56.875864] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:56.875869] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.875871] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.875873] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337656875826) [2024-09-26 16:00:56.888516] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15] success to notify_need_writing_throttling True [2024-09-26 16:00:56.888527] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.910956] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=22] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337656840735]}) [2024-09-26 16:00:56.918778] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:56.918788] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.919839] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:56.919941] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:56.920563] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=503312, clean_num=31457) [2024-09-26 16:00:56.949047] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21] success to notify_need_writing_throttling True [2024-09-26 16:00:56.949065] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.976443] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:56.976474] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337656976438) [2024-09-26 16:00:56.976481] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337656875826, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:56.976501] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223386212852, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:56.976509] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:56.976518] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:56.976524] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:56.976528] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:56.976534] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.976537] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:56.976539] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337656976484) [2024-09-26 16:00:56.979354] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=38] success to notify_need_writing_throttling True [2024-09-26 16:00:56.979385] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:56.984293] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:56.985174] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] tx gc loop thread is running(MTL_ID()=1) [2024-09-26 16:00:56.985182] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] try gc retain ctx [2024-09-26 16:00:56.985565] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-09-26 16:00:56.985591] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=24] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:57.010577] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:57.010587] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.011563] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337656840735]}) [2024-09-26 16:00:57.022437] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592A-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:57.022446] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592A-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:57.022471] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592A-0-0] [lt=24][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:57.022476] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592A-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:57.032236] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500821, support_ssl=false) [2024-09-26 16:00:57.032267] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221500821, sess.client_addr_="127.0.0.1:36374") [2024-09-26 16:00:57.032273] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500821, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:57.032315] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:57.032324] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:57.032364] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA3-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:57.032565] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA3-0-0] [lt=3][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:57.032573] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA3-0-0] [lt=7][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"Z8JJs%\#4}:G9%K.?6.%"}, ret=-4043) [2024-09-26 16:00:57.032590] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA3-0-0] [lt=15][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:57.032594] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA3-0-0] [lt=4][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:57.032602] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA3-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:57.032635] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA3-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500821, proxy_sessid:0}) [2024-09-26 16:00:57.032641] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA3-0-0] [lt=5] mark session id unused(sessid=3221500821) [2024-09-26 16:00:57.032645] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA3-0-0] [lt=2][errcode=0] server close connection(sessid=3221500821, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:57.032649] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA3-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500821, proxy_sessid=0) [2024-09-26 16:00:57.032651] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA3-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:00:57.032657] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA3-0-0] [lt=5] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500821, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:57.032688] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500821, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337657032058, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:57.032699] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221500821, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:57.032702] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500821, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337657032058, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:57.032706] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500821, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:57.040865] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:57.040871] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.058727] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85892-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:57.058736] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85892-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:57.058764] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85892-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:57.067328] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=8] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:57.067637] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7C-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:57.067647] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7C-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:57.067668] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7C-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:57.071215] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:57.071221] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.077016] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:57.077024] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:57.077031] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337657077014) [2024-09-26 16:00:57.077037] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337656976484, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:57.077043] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:57.077051] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:57.077054] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:57.077326] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885770-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:57.077367] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885770-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:57.077393] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337657077392, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=345, wlock_time=9, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:57.077401] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:57.077417] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223386313618, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:57.077422] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:57.077428] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:57.077433] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:57.077436] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:57.077441] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.077443] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.077445] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337657077406) [2024-09-26 16:00:57.087747] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=8] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:57.101459] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:57.101469] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.101492] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:57.112207] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337656840735]}) [2024-09-26 16:00:57.119991] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:134, tid:3024}]) [2024-09-26 16:00:57.121070] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:57.121184] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=49, refresh_score_time=13, wash_time=2) [2024-09-26 16:00:57.121806] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=534769, clean_num=31457) [2024-09-26 16:00:57.131679] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:57.131685] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.134615] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10790.00) [2024-09-26 16:00:57.134622] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:57.134633] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:57.134637] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:57.161981] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:57.161989] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.177598] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:57.177608] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337657177595) [2024-09-26 16:00:57.177613] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337657077040, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:57.177630] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223386414072, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:57.177636] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:57.177643] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:57.177648] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:57.177651] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:57.177657] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.177659] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.177662] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337657177617) [2024-09-26 16:00:57.184484] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:57.188392] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337656187305]) [2024-09-26 16:00:57.188400] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337656187305], now0=[mts=1727337656187305], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:57.188407] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337656187243, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:57.188433] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=25][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:57.188441] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337656187243, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:57.188449] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337656187243, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:57.188466] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:57.188470] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:57.188473] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:57.188476] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:57.188478] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:57.188481] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=19, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:57.188485] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=19000, remain_us=9805847, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:57.193208] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:57.193215] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.207938] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:19, local_retry_times:19, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:57.207949] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:57.207951] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:57.207954] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:57.207960] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:57.207999] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:57.208002] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:57.209041] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:57.209052] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:57.209057] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:57.213762] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337656840735]}) [2024-09-26 16:00:57.213784] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D966-0-0] [lt=7][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337657213732}) [2024-09-26 16:00:57.224175] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:57.224182] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.253207] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:57.253223] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=14][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:57.253228] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:57.253232] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:57.253236] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:57.253238] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:57.253242] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:57.253244] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:57.253246] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:57.253248] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:57.253250] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:57.253252] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:57.253255] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:57.253257] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:57.253263] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:57.253265] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:57.253268] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:57.253270] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:57.253273] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:00:57.253275] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:57.253278] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:57.253283] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:57.253289] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:57.253291] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:57.253292] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:57.253298] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:57.253302] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:57.253305] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:00:57.253308] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:57.253310] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:57.253315] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=5][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:57.253318] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337657253135, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:57.253321] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:57.253324] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:00:57.253352] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=3][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:57.253355] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:00:57.253358] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:57.253360] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=1][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:57.253363] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:57.253366] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=3][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:00:57.253368] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A1-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:57.254235] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:57.254244] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.278230] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:57.278247] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337657278227) [2024-09-26 16:00:57.278256] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337657177617, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:57.278283] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=223386514680, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:57.278290] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:57.278298] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:57.278305] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:57.278308] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:57.278315] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.278317] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.278320] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337657278269) [2024-09-26 16:00:57.285022] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:57.285030] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.314330] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337656840735]}) [2024-09-26 16:00:57.314352] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=15][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337658243537, now=[mts=1727337656313808]) [2024-09-26 16:00:57.314358] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337658243537, now=[mts=1727337656313808], now0=[mts=1727337656313808], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:57.314375] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=16][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269025820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337656313191, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:57.314411] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=35][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269025820}) [2024-09-26 16:00:57.314420] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269025820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337656313191, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:57.314427] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269025820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337656313191, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337656313779, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269025820}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:57.314444] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:57.314450] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:57.314452] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:57.314455] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:57.314458] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:57.314461] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:57.314466] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=999072, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337658313537) [2024-09-26 16:00:57.314469] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:57.314476] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:57.314479] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:57.314480] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:57.314486] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:57.314534] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:00:57.314540] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:57.316343] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:57.316351] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.322267] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:57.322372] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=50, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:57.322978] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=566226, clean_num=31457) [2024-09-26 16:00:57.341348] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.341359] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337657341345], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:57.341366] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337657341345], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:57.349012] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:57.349020] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.379786] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:57.379794] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.379796] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:57.379807] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337657379793) [2024-09-26 16:00:57.379812] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337657278268, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:57.379830] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223386616239, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:57.379838] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:57.379847] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:57.379852] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:57.379855] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:57.379862] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.379865] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.379869] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337657379815) [2024-09-26 16:00:57.384795] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=5] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:57.390511] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=6] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:57.392475] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=17] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:57.411059] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:57.411068] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.414569] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337657341345]}) [2024-09-26 16:00:57.416914] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D40F6-0-0] [lt=11][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:57.441183] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:57.441190] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.443247] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337657443246, bytes: 14694718, bw: 0.003620 MB/s, add_ts: 1006963, add_bytes: 3822 [2024-09-26 16:00:57.463787] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.463798] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.463803] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.463808] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.463810] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.463812] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.464756] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.464766] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.464783] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.464789] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.464791] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.464796] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.465718] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.465727] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.465741] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=14][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.465747] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.465750] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.465751] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.466393] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.466398] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.466401] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.466404] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.466406] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.466407] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.467661] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.467667] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.467669] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.467682] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.467684] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.467685] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.468651] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.468659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.468675] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=15][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.468681] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.468684] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.468689] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.469646] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.469651] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.469653] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.469667] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.469696] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=29][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.469698] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.470232] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.470238] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.470240] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.470243] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.470245] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.470247] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:57.471581] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.471588] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.471609] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:57.471614] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.472418] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.472427] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.473526] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.473542] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.474529] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.474535] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.476552] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.476559] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.476610] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] ====== tenant freeze timer task ====== [2024-09-26 16:00:57.476617] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:57.476657] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337655631632, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337655631601, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337655631649, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:57.476689] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:57.476757] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E1-0-0] [lt=2] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:57.476763] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E1-0-0] [lt=4] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:57.476767] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E1-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:57.476772] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E1-0-0] [lt=4] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:57.476775] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E1-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:57.476778] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E1-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:57.476781] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E1-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:57.476791] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E1-0-0] [lt=9] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:57.476794] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E1-0-0] [lt=3][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:57.476797] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E1-0-0] [lt=3] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:57.476799] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E1-0-0] [lt=1][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:57.476834] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851E1-0-0] [lt=3] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:57.477109] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.477120] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.478458] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.478466] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.480478] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.480483] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.480480] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:57.480489] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337657480477) [2024-09-26 16:00:57.480494] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337657379815, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:57.480512] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223386716950, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:57.480518] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:57.480526] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:57.480531] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:57.480533] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:57.480539] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.480542] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.480544] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337657480498) [2024-09-26 16:00:57.481346] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.481354] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.481949] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.481956] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.483317] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.483323] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.483851] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.483889] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.484524] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.484531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.485239] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.485250] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.486242] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.486248] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.487302] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.487307] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.488178] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.488185] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.490153] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.490172] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.491172] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.491177] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.492112] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.492121] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.492925] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.492934] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.494070] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.494074] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.495022] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.495026] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.495970] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.495993] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.496968] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.496973] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.498019] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.498026] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.499920] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.499927] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.500882] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.500888] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.501885] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.501891] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.501910] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:57.501916] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.502827] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.502833] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.503811] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.503820] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.504459] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.504467] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.505772] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.505779] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.506793] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.506798] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.507721] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.507727] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.508745] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.508752] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.509680] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.509690] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.510656] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.510663] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.511650] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.511656] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.512169] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.512176] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.513572] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.513587] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.514262] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.514269] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:57.515005] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337657341345]}) [2024-09-26 16:00:57.523442] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:57.523562] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=50, refresh_score_time=31, wash_time=2) [2024-09-26 16:00:57.524179] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=597683, clean_num=31457) [2024-09-26 16:00:57.529895] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592B-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:57.529908] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592B-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:57.529918] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592B-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:57.529923] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592B-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:57.532083] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:57.532093] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.546778] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6587, mem_used=12478464) [2024-09-26 16:00:57.562438] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:57.562449] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.565334] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:00:57.581949] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:57.581976] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337657581945) [2024-09-26 16:00:57.581983] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337657480498, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:57.581998] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:57.582009] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:57.582012] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:57.582356] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:57.582365] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:57.582368] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:57.582371] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:57.582376] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=4][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:57.582378] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:57.582382] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:57.582384] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:57.582386] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:57.582388] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:57.582390] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:57.582392] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:57.582395] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:57.582397] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:57.582403] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:57.582407] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:57.582410] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:57.582412] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:57.582415] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:57.582417] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:57.582420] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:57.582425] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:57.582430] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:57.582432] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:57.582433] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:57.582440] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:57.582444] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:57.582446] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885771-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:57.582448] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:57.582451] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:57.582453] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:57.582455] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337657582278, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:57.582458] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:57.582461] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:57.582496] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:57.582500] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337657582498, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=494, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:57.582507] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:57.582510] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:57.582527] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223386818480, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:57.582533] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:57.582539] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:57.582543] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:57.582545] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:57.582551] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.582553] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.582554] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337657582514) [2024-09-26 16:00:57.586257] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] Cache replace map node details(ret=0, replace_node_count=1, replace_time=0, replace_start_pos=849312, replace_num=15728) [2024-09-26 16:00:57.586269] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=1, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:57.592693] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:00:57.592706] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.601489] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.601500] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.601514] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.602460] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.615568] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337657341345]}) [2024-09-26 16:00:57.618156] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855F1-0-0] [lt=7] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:57.622962] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:57.622970] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.657172] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:57.657524] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=350][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.663013] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:45) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=0] dump dtl interm result cost(us)(dump_cost=13633, ret=0, interm count=0, dump count=345) [2024-09-26 16:00:57.663977] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:57.676623] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:60) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11] clear dtl interm result cost(us)(clear_cost=13597, ret=0, expire_keys_.count()=0, dump count=345, interm count=0, clean count=0) [2024-09-26 16:00:57.677032] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=25] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:57.677052] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=19][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:57.677058] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:57.677062] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:57.677067] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:57.677069] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:57.677073] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:57.677076] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:57.677081] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:57.677083] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:57.677085] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:57.677087] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:57.677090] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:57.677093] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:57.677101] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:57.677103] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:57.677107] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:57.677109] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:57.677113] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:57.677116] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:57.677119] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:57.677125] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:57.677131] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:57.677134] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:57.677135] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:57.677142] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:57.677146] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD6-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:57.677149] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2458][ServerGTimer][T0][YB42C0A8002D-0006230056A84DD6-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:57.677152] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:57.677155] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:57.677158] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:57.677160] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337657676965, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:57.677164] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:57.677220] WDIAG [SHARE] refresh (ob_all_server_tracer.cpp:568) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] fail to get servers_info(ret=-5019, ret="OB_TABLE_NOT_EXIST", GCTX.sql_proxy_=0x55c7486d2e88) [2024-09-26 16:00:57.677222] WDIAG [SHARE] runTimerTask (ob_all_server_tracer.cpp:626) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to refresh all server map(ret=-5019) [2024-09-26 16:00:57.677536] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:57.682559] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:57.682583] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337657682556) [2024-09-26 16:00:57.682590] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337657581986, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:57.682609] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223386919018, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:57.682615] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:57.682623] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:57.682629] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:57.682631] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:57.682637] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.682640] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.682645] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337657682593) [2024-09-26 16:00:57.688043] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:57.688054] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.715768] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337657341345]}) [2024-09-26 16:00:57.718221] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:57.718229] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.724523] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:57.724627] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=74, refresh_score_time=16, wash_time=1) [2024-09-26 16:00:57.725218] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=629140, clean_num=31457) [2024-09-26 16:00:57.737141] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=9] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:00:57.751747] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:57.751758] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.757750] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:57.782192] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:57.782202] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.783146] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:57.783163] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337657783142) [2024-09-26 16:00:57.783169] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337657682593, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:57.783189] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223387018663, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:57.783194] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:57.783203] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:57.783207] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:57.783210] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:57.783216] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.783218] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.783220] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337657783173) [2024-09-26 16:00:57.786759] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=4] Cache replace map node details(ret=0, replace_node_count=1, replace_time=0, replace_start_pos=865040, replace_num=15728) [2024-09-26 16:00:57.786768] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=1, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:57.812261] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:57.812270] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.816347] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337657341345]}) [2024-09-26 16:00:57.821237] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.821274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.821279] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.821286] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.821290] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.821294] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.821588] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.822209] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.822213] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.822223] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.822230] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:57.842868] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:57.842878] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337657842865], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:57.842885] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337657842865], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:57.842888] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:57.842893] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:57.843151] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:57.843159] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.873966] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:57.873975] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.881878] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=13] PNIO [ratelimit] time: 1727337657881877, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007073, add_bytes: 0 [2024-09-26 16:00:57.883764] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:57.883774] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337657883760) [2024-09-26 16:00:57.883779] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337657783173, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:57.883796] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223387120252, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:57.883802] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:57.883809] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:57.883814] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:57.883817] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:57.883822] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.883824] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.883826] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337657883782) [2024-09-26 16:00:57.889627] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=5][errcode=-4018] try free send task failed(ret=-4018) [2024-09-26 16:00:57.904243] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:57.904249] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.916953] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=34] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337657842865]}) [2024-09-26 16:00:57.925845] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:57.925972] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=15, wash_time=2) [2024-09-26 16:00:57.926560] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=5, clean_time=0, clean_start_pos=660597, clean_num=31457) [2024-09-26 16:00:57.934496] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:57.934505] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.964819] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:57.964826] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:57.984350] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:57.984359] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337657984348) [2024-09-26 16:00:57.984364] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337657883782, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:57.984391] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=223387220851, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:57.984397] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:57.984404] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:57.984417] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:57.984421] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:57.984427] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.984429] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:57.984430] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337657984378) [2024-09-26 16:00:57.987297] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:57.987631] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=4] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=880768, replace_num=15728) [2024-09-26 16:00:57.987639] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:00:57.995097] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:57.995102] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.017547] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337657842865]}) [2024-09-26 16:00:58.021589] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592C-0-0] [lt=34][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:58.021601] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592C-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:58.021610] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592C-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:58.021615] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592C-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:58.025367] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:58.025401] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=32][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.033387] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500825, support_ssl=false) [2024-09-26 16:00:58.033411] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221500825, sess.client_addr_="127.0.0.1:36376") [2024-09-26 16:00:58.033426] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f9262604030, session_id:3221500825, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:58.033481] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:58.033494] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=6] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:58.033553] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA5-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:58.033728] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA5-0-0] [lt=8][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:58.033737] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA5-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"s,zG2bz_K;%Fq'llXV'["}, ret=-4043) [2024-09-26 16:00:58.033744] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA5-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:58.033748] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA5-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:58.033755] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA5-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:58.033778] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA5-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500825, proxy_sessid:0}) [2024-09-26 16:00:58.033782] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA5-0-0] [lt=4] mark session id unused(sessid=3221500825) [2024-09-26 16:00:58.033802] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA5-0-0] [lt=17][errcode=0] server close connection(sessid=3221500825, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:58.033806] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA5-0-0] [lt=4][errcode=-4018] get session fail(ret=-4018, sessid=3221500825, proxy_sessid=0) [2024-09-26 16:00:58.033810] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA5-0-0] [lt=4][errcode=-4016] session is null [2024-09-26 16:00:58.033813] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA5-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500825, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:58.033870] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f9262604030, session_id:3221500825, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337658033207, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:58.033878] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5] kill and revert session(conn.sessid_=3221500825, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:58.033881] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9262604030, session_id:3221500825, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337658033207, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:58.033886] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500825, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:58.056636] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:00:58.056647] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.059875] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:58.059900] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=23][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:00:58.059907] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=6][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:00:58.059910] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:00:58.059915] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:58.059918] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:58.059922] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:00:58.059924] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:58.059927] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:58.059929] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:58.059931] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:58.059937] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:58.059940] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:58.059942] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:58.059949] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:58.059951] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.059954] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.059958] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:58.059961] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:00:58.059964] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:58.059966] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:58.059972] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:58.059978] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:58.059980] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:58.059982] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:58.059988] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:58.059992] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85893-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:58.059995] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85893-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:00:58.059998] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:58.060001] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85893-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:58.060003] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:58.060006] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337658059803, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:00:58.060009] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85893-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:58.060012] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85893-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:00:58.060039] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85893-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:00:58.060042] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85893-0-0] [lt=3][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:00:58.067934] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=9] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:58.068239] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:58.068249] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:58.068253] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:58.068256] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:58.068262] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=5][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:58.068264] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:58.068267] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:58.068269] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:58.068271] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:58.068272] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:58.068274] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:58.068276] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:58.068285] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:58.068290] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=5][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:58.068296] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:58.068298] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.068301] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.068303] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:58.068305] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:00:58.068307] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:58.068310] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:58.068314] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:58.068321] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:58.068323] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:58.068324] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:58.068329] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:58.068332] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D7E-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:58.068336] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D7E-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:00:58.068339] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:58.068342] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:58.068344] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:58.068347] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337658068191, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:00:58.068351] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:58.068397] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:00:58.084614] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:58.084622] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:58.084628] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337658084611) [2024-09-26 16:00:58.084633] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337657984377, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:58.084638] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:58.084646] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:58.084648] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:58.084925] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:58.084933] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:58.084936] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:58.084939] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:58.084941] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:58.084943] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:58.084946] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:58.084948] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:58.084949] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:58.084951] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:58.084952] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:58.084954] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:58.084966] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885772-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:58.084989] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337658084989, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=347, wlock_time=8, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:58.084997] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:58.085012] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223387321090, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:58.085016] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:58.085022] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:58.085026] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:58.085028] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:58.085033] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.085035] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.085037] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337658085000) [2024-09-26 16:00:58.086901] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:58.086911] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.087825] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:58.117188] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:58.117198] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.117207] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:58.118198] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337657842865]}) [2024-09-26 16:00:58.127073] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:58.127175] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:58.127780] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=692054, clean_num=31457) [2024-09-26 16:00:58.138764] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 579.00) [2024-09-26 16:00:58.138773] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:58.138784] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:58.138788] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:58.147463] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:58.147472] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.177805] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:58.177814] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.184669] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:58.184680] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337658184666) [2024-09-26 16:00:58.184685] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337658084635, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:58.184705] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223387421042, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:58.184711] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:58.184718] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:58.184724] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:58.184726] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:58.184732] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.184733] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.184735] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337658184688) [2024-09-26 16:00:58.187776] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:00:58.201350] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:00:58.201361] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=9] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:00:58.201370] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:00:58.208694] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=53][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337657207998]) [2024-09-26 16:00:58.208704] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337657207998], now0=[mts=1727337657207998], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:58.208720] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:58.208711] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337657207901, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:58.208727] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.208728] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:58.208736] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337657207901, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:58.208744] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337657207901, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:58.208761] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:58.208766] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:58.208769] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:58.208773] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:58.208775] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:58.208780] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=20, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:58.208787] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5] will sleep(sleep_us=20000, remain_us=8785546, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:58.218777] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337657842865]}) [2024-09-26 16:00:58.218802] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D972-0-0] [lt=5][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337658218768}) [2024-09-26 16:00:58.229493] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:20, local_retry_times:20, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:58.229507] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:58.229511] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:58.229512] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:58.229519] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:58.229559] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:58.229563] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:58.239264] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:58.239281] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.254269] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:58.254284] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=14][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:00:58.254289] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:58.254306] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:00:58.254310] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:58.254312] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:58.254320] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:00:58.254322] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:58.254324] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:58.254326] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:58.254328] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:58.254331] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=3][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:58.254352] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A2-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:58.269549] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:58.269560] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.284705] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:58.284737] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337658284700) [2024-09-26 16:00:58.284745] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337658184688, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:58.284763] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223387520668, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:58.284769] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:58.284777] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:58.284783] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:58.284785] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:58.284791] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.284794] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.284796] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337658284748) [2024-09-26 16:00:58.299823] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:58.299832] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.315485] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337658243537, now=[mts=1727337657314533]) [2024-09-26 16:00:58.315495] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337658243537, now=[mts=1727337657314533], now0=[mts=1727337657314533], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:58.315502] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269025820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337657314319, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:58.315519] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269025820}) [2024-09-26 16:00:58.315527] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269025820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337657314319, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:58.315534] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269025820, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337657314319, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337656313779, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269025820}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:58.315552] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:58.315556] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:58.315559] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:58.315562] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:58.315564] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:58.315568] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:58.315572] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:00:58.315578] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:58.315580] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:58.315581] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:00:58.315586] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:00:58.315590] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2001810) [2024-09-26 16:00:58.315592] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:00:58.315595] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:58.315598] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:00:58.315600] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:00:58.315602] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:00:58.315607] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:00:58.315635] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:00:58.315638] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:00:58.315640] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:00:58.315642] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:58.315646] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:00:58.315648] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:00:58.315769] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE0-0-0] [lt=3][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:00:58.315778] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:00:58.315780] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:00:58.315796] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:58.315800] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:00:58.316074] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:00:58.316081] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=7][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:58.319373] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD7-0-0] [lt=2][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-09-26 16:00:58.319456] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD7-0-0] [lt=83][errcode=-4002] refresh sever list failed(ret=-4002) [2024-09-26 16:00:58.319459] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD7-0-0] [lt=3][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-09-26 16:00:58.319471] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337657842865]}) [2024-09-26 16:00:58.325369] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:53, tid:2919}]) [2024-09-26 16:00:58.328275] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:58.328377] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=62, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:58.328983] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=723511, clean_num=31457) [2024-09-26 16:00:58.331141] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:58.331148] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.343892] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:58.343899] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337658343889], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:58.343905] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337658343889], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:58.361360] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:58.361366] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.384886] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:58.384905] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337658384883) [2024-09-26 16:00:58.384910] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337658284748, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:58.384926] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223387621287, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:58.384931] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:58.384938] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:58.384942] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:58.384944] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:58.384949] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.384952] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.384954] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337658384913) [2024-09-26 16:00:58.388707] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:00:58.390653] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=10] current monitor number(seq_=-1) [2024-09-26 16:00:58.390694] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=9] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:58.391650] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:58.391656] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.416362] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:58.416373] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=6] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:58.416380] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:58.416384] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:58.416388] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:58.416391] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:00:58.416393] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:58.416396] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:58.416399] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:58.416402] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:58.416404] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:58.416406] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:58.416409] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:58.416411] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:58.416416] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=5][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:58.416419] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:58.416425] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:58.416428] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.416431] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.416434] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:58.416438] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-09-26 16:00:58.416444] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:58.416449] WDIAG [SERVER] do_process (obmp_query.cpp:771) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=5][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-09-26 16:00:58.416455] WDIAG [SERVER] do_process (obmp_query.cpp:903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, session={this:0x7f92a230bdf8, id:3221487638, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-09-26 16:00:58.416466] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F53-0-0] [lt=6] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:58.417333] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D4140-0-0] [lt=25][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:58.420004] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337658343889]}) [2024-09-26 16:00:58.421932] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:58.421939] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.443421] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337658443421, bytes: 14698258, bw: 0.003375 MB/s, add_ts: 1000175, add_bytes: 3540 [2024-09-26 16:00:58.452189] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:58.452197] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.453334] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:33) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=3] === Run print tenant memory usage task === [2024-09-26 16:00:58.453378] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=6] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 413,495,296 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 180,654,080 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 88,080,384 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 12,582,912 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 17,969,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 11,292,672 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 92,430,336 limit= 9,223,372,036,854,775,807 [2024-09-26 16:00:58.453480] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:176) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=4] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 180,654,080 used= 173,086,912 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] hold= 33,574,912 used= 33,554,464 count= 1 avg_used= 33,554,464 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 18,894,848 used= 18,874,392 count= 1 avg_used= 18,874,392 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 12,460,032 used= 11,125,608 count= 169 avg_used= 65,832 block_cnt= 169 chunk_cnt= 29 mod=[T]ObSessionDIB [MEMORY] hold= 10,776,576 used= 10,735,904 count= 11 avg_used= 975,991 block_cnt= 11 chunk_cnt= 7 mod=IoControl [MEMORY] hold= 8,585,216 used= 8,566,048 count= 1 avg_used= 8,566,048 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 5,251,072 used= 5,238,784 count= 32 avg_used= 163,712 block_cnt= 32 chunk_cnt= 9 mod=Iteratorget_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=1, size_used=6596, mem_used=12478464) [2024-09-26 16:00:58.561065] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] LogLoopThread round_cost_time(us)(round_cost_time=113) [2024-09-26 16:00:58.565466] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=9] gc stale ls task succ [2024-09-26 16:00:58.573354] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:58.573363] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.586017] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:58.586030] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337658586013) [2024-09-26 16:00:58.586036] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337658485484, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:58.586042] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:58.586053] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:58.586056] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:58.586375] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:58.586388] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:58.586393] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:58.586396] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:58.586400] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:58.586402] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:58.586405] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:58.586408] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:58.586410] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:58.586412] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:58.586414] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:58.586417] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:58.586420] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:58.586442] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=21][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:58.586450] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:58.586453] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.586475] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=21][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.586480] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=5][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:58.586483] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:58.586486] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:58.586489] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:58.586495] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:58.586502] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:58.586505] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:58.586506] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:58.586513] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:58.586517] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:58.586520] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885773-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:58.586523] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:58.586527] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:58.586529] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:58.586532] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337658586298, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:58.586536] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:58.586538] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:58.586569] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:58.586575] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337658586574, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=527, wlock_time=10, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:58.586582] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:58.586585] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:58.586602] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223387822512, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:58.586608] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:58.586614] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:58.586619] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:58.586622] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:58.586628] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.586630] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.586633] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337658586589) [2024-09-26 16:00:58.589906] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:00:58.602604] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.602618] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.602622] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.602631] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.603581] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:58.603586] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.616481] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:783) [2618][AutoLSLocRpc][T0][YB42C0A8002D-0006230057E85894-0-0] [lt=14] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.0.45:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, location:{server:"192.168.0.45:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:374}}]) [2024-09-26 16:00:58.622146] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337658343889]}) [2024-09-26 16:00:58.634870] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:58.634878] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.664139] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:58.665103] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:58.665110] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.683022] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:58.683032] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:00:58.683036] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:00:58.683039] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:00:58.683042] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:58.683044] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:58.683047] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:00:58.683050] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:58.683052] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:58.683053] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:58.683055] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:58.683058] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:58.683060] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:58.683062] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:58.683068] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:58.683070] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.683073] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.683075] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:58.683077] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, ret=-5019) [2024-09-26 16:00:58.683079] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:58.683082] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:58.683086] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:58.683092] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:58.683093] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:58.683095] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:58.683100] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:58.683103] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:58.683106] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD7-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-09-26 16:00:58.683108] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:58.683110] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:58.683112] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:58.683115] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337658682961, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:58.683118] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:58.683120] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f92b4056738, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, columns_str="zone") [2024-09-26 16:00:58.683125] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:00:58.683150] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:634) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f92b4056738, where_condition="where svr_ip='192.168.0.45' and svr_port=2882", zone_name_holder=) [2024-09-26 16:00:58.683153] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:567) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:58.683156] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:576) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:00:58.683158] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:58.683163] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:58.686623] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:58.686635] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337658686619) [2024-09-26 16:00:58.686640] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337658586039, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:58.686656] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223387922167, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:58.686662] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:58.686670] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:58.686675] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:58.686678] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:58.686683] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.686685] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.686687] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337658686643) [2024-09-26 16:00:58.695464] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:58.695485] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.722747] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337658343889]}) [2024-09-26 16:00:58.725678] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:58.725683] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.730717] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:58.730831] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=53, refresh_score_time=24, wash_time=2) [2024-09-26 16:00:58.731460] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=786425, clean_num=31457) [2024-09-26 16:00:58.755952] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:58.755967] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.761824] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:58.761840] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1073) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=5] replay status is not follower(min_unreplayed_lsn={lsn:629872191939}, base_lsn={lsn:627966418944}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:00:58.761862] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=22] get_replay_process success(id={id:1}, replayed_log_size=1905772995, unreplayed_log_size=0) [2024-09-26 16:00:58.761875] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:159) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=3] dump tenant replay process(tenant_id=1, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=1817, last_replayed_log_size(MB)=1817, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-09-26 16:00:58.786221] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:58.786228] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.787192] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:58.787220] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337658787190) [2024-09-26 16:00:58.787226] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337658686642, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:58.787242] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223388022728, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:58.787248] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:58.787254] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:58.787259] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:58.787261] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:58.787267] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.787269] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.787271] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337658787229) [2024-09-26 16:00:58.790139] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:00:58.817513] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:58.817520] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.821420] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.821421] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=36] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.821430] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.821435] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.821440] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.821445] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.821467] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.821479] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.822402] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.822422] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.822430] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:58.823422] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337658343889]}) [2024-09-26 16:00:58.844960] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:58.844971] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337658844957], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:58.844978] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337658844957], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:58.844982] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:58.844985] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:58.847757] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:58.847778] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.877102] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [2910][T1_RFLWorker][T1][YB42C0A8002D-0006230055684B24-0-0] [lt=6] ObRemoteFetchWorker is running(thread_index=0) [2024-09-26 16:00:58.878029] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:58.878044] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.888828] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:58.888837] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337658888824) [2024-09-26 16:00:58.888842] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337658787229, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:58.888856] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223388125301, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:58.888862] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:58.888871] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:58.888876] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:58.888878] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:58.888887] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.888889] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.888891] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337658888845) [2024-09-26 16:00:58.889736] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337658889735, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007858, add_bytes: 0 [2024-09-26 16:00:58.909280] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:58.909325] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.922019] INFO [CLOG] run1 (ob_garbage_collector.cpp:1338) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] Garbage Collector is running(seq_=346, gc_interval=10000000) [2024-09-26 16:00:58.922047] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1473) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] self is leader, skip it(ls->get_ls_id()={id:1}) [2024-09-26 16:00:58.922059] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1431) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] gc_check_member_list_ cost time(ret=0, time_us=31) [2024-09-26 16:00:58.922063] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] execute_gc cost time(ret=0, time_us=0) [2024-09-26 16:00:58.922067] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=2] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-09-26 16:00:58.922071] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2024-09-26 16:00:58.922074] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] execute_gc cost time(ret=0, time_us=3) [2024-09-26 16:00:58.922076] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=2] ObSafeDestroyHandler start process [2024-09-26 16:00:58.922079] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2024-09-26 16:00:58.922087] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2024-09-26 16:00:58.923942] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337658844957]}) [2024-09-26 16:00:58.931894] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:58.931993] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=48, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:58.932595] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=817882, clean_num=31457) [2024-09-26 16:00:58.940524] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:00:58.940532] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.963326] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-09-26 16:00:58.963348] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=20][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-09-26 16:00:58.963353] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-09-26 16:00:58.963357] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-09-26 16:00:58.963360] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:58.963362] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:58.963366] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-09-26 16:00:58.963369] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:58.963371] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:58.963373] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:58.963375] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:58.963377] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:58.963379] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:58.963381] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:58.963388] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:58.963390] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.963394] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.963396] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:58.963398] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:58.963401] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:58.963403] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:58.963408] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:58.963414] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:58.963416] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:58.963417] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:58.963424] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:58.963428] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:58.963431] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2024-09-26 16:00:58.963434] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:58.963436] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:58.963438] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:58.963440] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337658963252, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:58.963443] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:58.963447] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1111) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:00:58.963472] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:801) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:00:58.963475] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:67) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=3][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:58.963478] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:436) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:58.963480] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:95) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:58.963481] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:109) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=1] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:58.963708] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-09-26 16:00:58.963714] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=5][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2024-09-26 16:00:58.963716] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2024-09-26 16:00:58.963718] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2024-09-26 16:00:58.963720] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:58.963722] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:58.963724] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_tenant' doesn't exist [2024-09-26 16:00:58.963726] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:58.963727] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:58.963729] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:58.963730] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:58.963732] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:58.963733] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:58.963734] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:58.963736] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:58.963739] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.963741] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:58.963743] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:58.963744] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:58.963746] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:58.963747] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:58.963750] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:58.963753] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:58.963755] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:58.963756] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:58.963759] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:58.963761] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:58.963763] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, aret=-5019, ret=-5019) [2024-09-26 16:00:58.963765] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:00:58.963767] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:58.963769] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:58.963771] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337658963684, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:00:58.963774] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:58.963775] WDIAG [SHARE] read_tenants (ob_unit_table_operator.cpp:1174) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] execute sql failed(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:58.963786] WDIAG [SHARE] get_tenants (ob_unit_table_operator.cpp:108) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] fail read tenants(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:00:58.963788] WDIAG [SHARE] get_tenants (ob_unit_getter.cpp:198) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=1][errcode=-5019] ut_operator get_resource_pools failed(ret=-5019) [2024-09-26 16:00:58.963790] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:125) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2][errcode=-5019] get cluster tenants fail(ret=-5019) [2024-09-26 16:00:58.963801] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:325) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2] cal_all_part_disk_default_percentage succ(data_dir="/root/observer/store/sstable", clog_dir="/root/observer/store/clog", shared_mode=true, data_disk_total_size=99765694464, data_disk_default_percentage=60, clog_disk_total_size=99765694464, clog_disk_default_percentage=30) [2024-09-26 16:00:58.963805] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:361) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=4] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=99765694464, disk_size=15032385536) [2024-09-26 16:00:58.963808] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=3] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=99765694464) [2024-09-26 16:00:58.963812] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=2] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=99765694464, this={dir::"/root/observer/store/clog/log_pool", dir_fd:15, meta_fd:16, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:9388, max_block_id:9581, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2024-09-26 16:00:58.963816] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:133) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD8-0-0] [lt=4] refresh tenant config(tenants=[], ret=-5019) [2024-09-26 16:00:58.970844] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:58.970853] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:58.986478] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:58.989386] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:58.989435] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337658989374) [2024-09-26 16:00:58.989460] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337658888845, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:58.989476] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223388225852, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:58.989482] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:58.989489] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:58.989494] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:58.989497] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:58.989503] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.989506] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:58.989507] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337658989463) [2024-09-26 16:00:58.990350] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:00:59.001088] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.001096] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.025548] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337658844957]}) [2024-09-26 16:00:59.025857] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA5-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:59.025870] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA5-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-09-26 16:00:59.025874] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA5-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-09-26 16:00:59.025877] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA5-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-09-26 16:00:59.025881] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA5-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:59.025883] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA5-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:59.025886] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA5-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-09-26 16:00:59.025889] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA5-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:59.025890] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA5-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:59.025892] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA5-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:59.025893] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA5-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:59.025911] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA5-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:59.025935] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=0] ls blacklist refresh finish(cost_time=428) [2024-09-26 16:00:59.026100] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592E-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:59.026106] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592E-0-0] [lt=6][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:59.026125] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592E-0-0] [lt=18][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:59.026130] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592E-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:59.031356] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.031364] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.034517] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500830, support_ssl=false) [2024-09-26 16:00:59.034544] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221500830, sess.client_addr_="127.0.0.1:36378") [2024-09-26 16:00:59.034551] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500830, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:59.034572] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:00:59.034592] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:00:59.034625] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA4-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:00:59.034786] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA4-0-0] [lt=7][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:00:59.034796] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA4-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"ny8JL~VpS/-7i0Mv*k]r"}, ret=-4043) [2024-09-26 16:00:59.034803] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA4-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:00:59.034806] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA4-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:00:59.034816] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA4-0-0] [lt=3] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:59.034834] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA4-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500830, proxy_sessid:0}) [2024-09-26 16:00:59.034838] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA4-0-0] [lt=4] mark session id unused(sessid=3221500830) [2024-09-26 16:00:59.034841] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA4-0-0] [lt=2][errcode=0] server close connection(sessid=3221500830, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:00:59.034845] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA4-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500830, proxy_sessid=0) [2024-09-26 16:00:59.034849] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA4-0-0] [lt=4][errcode=-4016] session is null [2024-09-26 16:00:59.034854] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA4-0-0] [lt=5] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500830, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:00:59.034885] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500830, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337659034343, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:00:59.034892] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] kill and revert session(conn.sessid_=3221500830, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:00:59.034895] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500830, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337659034343, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:00:59.034899] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500830, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:00:59.060997] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85894-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:00:59.061009] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85894-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:59.061031] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85894-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:59.061646] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.061654] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.068501] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=9] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:00:59.068763] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D80-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:00:59.068773] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D80-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:59.068791] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D80-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:59.088012] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=8] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:00:59.090039] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:00:59.090049] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:59.090055] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337659090037) [2024-09-26 16:00:59.090059] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337658989463, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:59.090065] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:59.090073] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:59.090076] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:59.090343] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885774-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:59.090352] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885774-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:59.090386] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885774-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:59.090410] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337659090409, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=339, wlock_time=8, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:59.090418] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:59.090433] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223388326630, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:59.090437] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:59.090443] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:59.090449] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:59.090452] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:59.090457] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.090460] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.090461] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337659090422) [2024-09-26 16:00:59.091929] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:59.091936] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.122227] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:59.122242] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.122250] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:00:59.126124] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337658844957]}) [2024-09-26 16:00:59.133063] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:59.133164] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:00:59.133774] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=849339, clean_num=31457) [2024-09-26 16:00:59.144720] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 13406.00) [2024-09-26 16:00:59.144729] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:00:59.144741] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:59.144745] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:00:59.146624] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:221) [2596][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=9] [ASYNC TASK QUEUE](queue_size=0) [2024-09-26 16:00:59.152531] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:00:59.152556] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.155405] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-09-26 16:00:59.155717] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD7-0-0] [lt=26] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-09-26 16:00:59.155728] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD7-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:59.155750] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD7-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:59.182823] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.182830] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.190575] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:00:59.190592] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:59.190601] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337659190582) [2024-09-26 16:00:59.190605] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337659090062, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:59.190622] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223388427070, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:59.190628] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:59.190634] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:59.190639] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:59.190641] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:59.190646] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.190648] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.190650] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337659190608) [2024-09-26 16:00:59.209393] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:00:59.209405] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:00:59.209409] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:00:59.214041] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.214049] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.226765] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337658844957]}) [2024-09-26 16:00:59.226783] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D97E-0-0] [lt=10][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337659226760}) [2024-09-26 16:00:59.229082] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=33][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337658229557]) [2024-09-26 16:00:59.229091] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337658229557], now0=[mts=1727337658229557], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:59.229097] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337658228722, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:59.229116] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=18][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:00:59.229123] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337658228722, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:59.229131] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337658228722, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:59.229149] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:59.229154] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:59.229157] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:59.229162] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:59.229164] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:00:59.229168] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=21, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:59.229173] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] will sleep(sleep_us=21000, remain_us=7765159, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:00:59.244317] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.244325] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.251113] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:21, local_retry_times:21, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:59.251140] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=20][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:59.251144] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:59.251147] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:59.251155] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:59.251212] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:00:59.251217] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:59.255457] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A3-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:00:59.255472] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A3-0-0] [lt=13][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:59.255518] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A3-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:59.274591] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25] success to notify_need_writing_throttling True [2024-09-26 16:00:59.274605] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.290738] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:59.290763] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337659290733) [2024-09-26 16:00:59.290772] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337659190608, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:59.290796] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223388526867, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:59.290804] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:59.290813] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:59.290819] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:59.290823] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:59.290830] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.290832] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.290835] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337659290776) [2024-09-26 16:00:59.305826] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:00:59.305839] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.318476] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337660245803, now=[mts=1727337658316072]) [2024-09-26 16:00:59.318494] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=17][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337660245803, now=[mts=1727337658316072], now0=[mts=1727337658316072], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:00:59.318537] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=41][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926901c400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337658315833, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:59.318562] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=23][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926901c400}) [2024-09-26 16:00:59.318572] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=10][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926901c400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337658315833, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:00:59.318582] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=9] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926901c400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337658315833, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337658316043, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926901c400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:00:59.318605] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=22][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:00:59.318623] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=6][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:00:59.318627] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=4][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:00:59.318630] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:00:59.318634] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=4][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:00:59.318640] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:59.318646] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5] already timeout, do not need sleep(sleep_us=0, remain_us=997157, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337660315803) [2024-09-26 16:00:59.318651] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:00:59.318660] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:59.318662] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:00:59.318665] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:00:59.318673] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:00:59.318721] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:00:59.318724] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:59.327360] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337658844957]}) [2024-09-26 16:00:59.333909] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:59.334037] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Wash time detail, (compute_wash_size_time=64, refresh_score_time=30, wash_time=2) [2024-09-26 16:00:59.334679] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=880796, clean_num=31457) [2024-09-26 16:00:59.336051] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:59.336060] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.343049] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [2917][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] start to check and clear obj lock(ls->get_ls_id()={id:1}) [2024-09-26 16:00:59.345322] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.345332] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337659345320], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:59.345342] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337659345320], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:59.345359] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337659345320]}) [2024-09-26 16:00:59.366122] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:00:59.366130] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.390757] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:00:59.390774] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=6] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:00:59.391791] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [2645][Blacklist][T0][Y0-0000000000000000-0-0] [lt=7] blacklist_loop exec finished(cost_time=10, is_enabled=true, send_cnt=0) [2024-09-26 16:00:59.391808] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:59.391819] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337659391805) [2024-09-26 16:00:59.391825] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337659290776, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:59.391843] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223388626901, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:59.391861] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:59.391870] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:59.391875] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:59.391878] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:59.391885] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.391888] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.391889] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337659391827) [2024-09-26 16:00:59.396661] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:59.396667] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.420182] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D419E-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:00:59.426947] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.426956] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.427599] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=24] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337659345320]}) [2024-09-26 16:00:59.431861] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=11][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:206, tid:3001}]) [2024-09-26 16:00:59.441638] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=20] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:00:59.450406] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337659450385, bytes: 14702080, bw: 0.003620 MB/s, add_ts: 1006964, add_bytes: 3822 [2024-09-26 16:00:59.457217] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:59.457225] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.465057] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.465068] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.465073] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.465078] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.465080] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.465082] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.466012] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.466023] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.466029] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.466033] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.466036] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.466037] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.467032] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.467037] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.467040] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.467042] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.467044] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.467046] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.468029] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.468032] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.468034] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.468036] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.468041] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.468042] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.468963] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.468968] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.468970] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.468973] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.468975] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.468976] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.469957] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.469962] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.469964] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.469967] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.469969] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.469970] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.470904] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.470910] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.470913] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.470917] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.470918] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.470920] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.471872] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.471877] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.471879] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.471882] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.471884] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.471886] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:00:59.472851] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.472858] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.473377] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.473382] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.473991] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.473998] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.474755] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.474764] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.475846] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.475852] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.476750] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.476756] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.477726] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.477733] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.478473] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.478477] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.479695] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.479709] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.480692] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.480700] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.481258] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.481267] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.482628] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.482637] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.483636] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.483647] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.484565] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.484572] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.485572] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.485579] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.486156] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.486163] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.486705] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.486712] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.487485] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:59.487493] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.487603] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] ====== tenant freeze timer task ====== [2024-09-26 16:00:59.487614] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:00:59.487645] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337655631632, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337655631601, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337655631649, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:00:59.487668] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:00:59.487683] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.487688] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.487720] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E2-0-0] [lt=3] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:00:59.487731] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E2-0-0] [lt=8] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:00:59.487736] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E2-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:00:59.487740] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E2-0-0] [lt=3] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:00:59.487744] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E2-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:00:59.487746] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E2-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:00:59.487748] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E2-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:00:59.487757] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E2-0-0] [lt=9] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:00:59.487761] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E2-0-0] [lt=4][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:59.487763] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E2-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:59.487766] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E2-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:00:59.487796] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851E2-0-0] [lt=7] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:00:59.488477] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.488487] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.489487] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.489495] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.490469] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.490477] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.491405] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.491411] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.492053] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.492058] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.492058] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:59.492068] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337659492055) [2024-09-26 16:00:59.492073] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337659391827, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:59.492091] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223388727940, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:59.492096] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:59.492104] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:59.492109] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:59.492112] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:59.492118] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.492120] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.492122] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337659492076) [2024-09-26 16:00:59.493373] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.493395] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.494355] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.494374] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.495295] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.495303] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.496349] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.496356] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.497289] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.497295] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.497994] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.498000] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.499238] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.499256] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.500218] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.500226] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.501166] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.501172] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.502153] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.502159] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.503005] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.503013] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.504119] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.504123] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.505109] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.505115] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.506093] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.506099] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.506694] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.506700] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.508023] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.508032] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.508571] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:00:59.508578] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=7][errcode=-4038] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:59.509976] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:00:59.517772] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.517780] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.520716] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [2940][T1_ArcFetcher][T1][YB42C0A8002D-0006230055E84B24-0-0] [lt=7] ObArchiveFetcher is running(thread_index=0) [2024-09-26 16:00:59.526731] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592F-0-0] [lt=28][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:00:59.526740] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592F-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:59.526748] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592F-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:00:59.526753] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718592F-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:00:59.528544] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337659345320]}) [2024-09-26 16:00:59.532478] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4038, dropped:5, tid:2580}]) [2024-09-26 16:00:59.535388] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:59.535499] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=73, refresh_score_time=16, wash_time=1) [2024-09-26 16:00:59.536086] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=912253, clean_num=31457) [2024-09-26 16:00:59.547234] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6608, mem_used=12478464) [2024-09-26 16:00:59.548191] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.548200] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.565633] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=9] gc stale ls task succ [2024-09-26 16:00:59.578321] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:00:59.578350] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.591062] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:00:59.592085] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:59.592098] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337659592080) [2024-09-26 16:00:59.592103] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337659492076, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:59.592110] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:59.592120] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:00:59.592124] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:00:59.592474] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:00:59.592484] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:00:59.592488] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:00:59.592492] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:00:59.592496] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:59.592499] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:59.592502] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:00:59.592505] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:59.592508] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:59.592510] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:59.592512] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:59.592515] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:59.592518] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:59.592520] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:59.592527] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:59.592530] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:59.592533] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:59.592545] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:59.592548] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:00:59.592551] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:59.592553] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:59.592560] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:59.592566] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:59.592569] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:59.592571] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:59.592577] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:59.592581] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:59.592583] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885775-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:00:59.592587] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:59.592589] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:59.592592] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:59.592595] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337659592378, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:59.592599] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:59.592601] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:00:59.592633] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:59.592636] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337659592635, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=520, wlock_time=13, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:00:59.592644] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:00:59.592647] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:00:59.592665] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223388828705, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:59.592670] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:59.592676] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:59.592681] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:59.592686] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:59.592692] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.592693] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.592695] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337659592650) [2024-09-26 16:00:59.601601] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.601611] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.601627] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.601645] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.608397] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14] success to notify_need_writing_throttling True [2024-09-26 16:00:59.608405] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.628827] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337659345320]}) [2024-09-26 16:00:59.638915] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:00:59.638925] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.664336] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:00:59.669198] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:59.669207] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.686788] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:00:59.692632] WDIAG [PALF] convert_to_ts (scn.cpp:265) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2024-09-26 16:00:59.692648] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1727337659692631, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1727337659692631, force_self_check=false, weak_read_refresh_interval=100000) [2024-09-26 16:00:59.692664] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:59.692669] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337659692662) [2024-09-26 16:00:59.692673] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337659592106, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:59.692689] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223388929098, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:59.692694] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:59.692702] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:59.692706] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:59.692708] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:59.692715] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.692718] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.692727] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337659692676) [2024-09-26 16:00:59.699476] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.699485] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.729479] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337659345320]}) [2024-09-26 16:00:59.730735] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.730742] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.736357] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:59.736456] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=50, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:59.737091] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=943710, clean_num=31457) [2024-09-26 16:00:59.760970] INFO [STORAGE.TRANS] print_statistics (ob_xa_define.cpp:467) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=8] xa statistics(total_active_xa_ctx_count=0, total_standby_clearup_count=0, total_success_xa_start=0, total_failure_xa_start=0, total_success_xa_prepare=0, total_failure_xa_prepare=0, total_success_xa_1pc_commit=0, total_failure_xa_1pc_commit=0, total_success_xa_2pc_commit=0, total_failure_xa_2pc_commit=0, total_failure_xa_rollback=0, total_success_dblink_promotion=0, total_failure_dblink_promotion=0, total_success_dblink=0, total_failure_dblink=0) [2024-09-26 16:00:59.760988] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.760992] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.769776] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:00:59.783453] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=5] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-09-26 16:00:59.783463] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-09-26 16:00:59.783469] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=5] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=692) [2024-09-26 16:00:59.791232] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:00:59.791235] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:59.791242] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.793232] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:59.793248] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337659793229) [2024-09-26 16:00:59.793253] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337659692676, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:59.793276] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get_max_applied_scn(scn={val:1727114270763524000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:374, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:1727114270763524000, v:0}, max_applied_cb_scn_:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:59.793286] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get_min_unreplayed_log_info(lsn={lsn:629872191939}, scn={val:1727114270763524001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:00:59.793312] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1}, delta=223389029746, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:59.793319] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:59.793325] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:59.793331] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:59.793333] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:59.793339] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.793341] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.793343] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337659793256) [2024-09-26 16:00:59.821534] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.821544] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.821548] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.821552] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.821551] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.821557] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.821565] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.821571] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.821687] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:00:59.821693] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.821864] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.821875] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.821881] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:00:59.829567] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337659345320]}) [2024-09-26 16:00:59.846061] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:00:59.846070] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337659846058], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:59.846077] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337659846058], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:00:59.846081] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:59.846084] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:00:59.851870] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:59.851877] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.874629] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, table_name.ptr()="data_size:16, data:5F5F616C6C5F6D657267655F696E666F", ret=-5019) [2024-09-26 16:00:59.874639] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, ret=-5019) [2024-09-26 16:00:59.874643] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_merge_info, db_name=oceanbase) [2024-09-26 16:00:59.874647] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_merge_info) [2024-09-26 16:00:59.874650] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:00:59.874652] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:00:59.874657] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_merge_info' doesn't exist [2024-09-26 16:00:59.874660] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:00:59.874663] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:00:59.874665] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:00:59.874667] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:00:59.874670] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:00:59.874673] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:00:59.874675] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:00:59.874682] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:00:59.874684] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:59.874687] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:00:59.874689] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:00:59.874691] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_merge_info WHERE tenant_id = '1', ret=-5019) [2024-09-26 16:00:59.874694] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:00:59.874697] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:00:59.874703] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:00:59.874709] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:59.874711] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:00:59.874713] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:00:59.874719] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, record_ret=-5019, ret=-5019) [2024-09-26 16:00:59.874724] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=4][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:00:59.874727] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508A-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, aret=-5019, ret=-5019) [2024-09-26 16:00:59.874729] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:59.874732] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:00:59.874736] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:00:59.874739] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337659874559, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:59.874742] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:00:59.874745] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:49) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:00:59.874774] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:890) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] failed to load global merge info(ret=-5019, ret="OB_TABLE_NOT_EXIST", global_merge_info={tenant_id:1, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2024-09-26 16:00:59.874785] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1005) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] fail to refresh merge info(tmp_ret=-5019, tmp_ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:00:59.875007] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:00:59.875014] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=7][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:00:59.882250] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.882259] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.893825] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:59.893835] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337659893822) [2024-09-26 16:00:59.893840] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337659793256, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:59.893858] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223389129462, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:59.893864] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:59.893872] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:59.893877] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:59.893880] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:59.893886] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.893889] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.893891] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337659893844) [2024-09-26 16:00:59.897783] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337659897782, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008047, add_bytes: 0 [2024-09-26 16:00:59.912415] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:00:59.912420] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.920137] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] ====== checkpoint timer task ====== [2024-09-26 16:00:59.920547] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] succ to get rec scn(*this={this:0x7f92a6a04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1726077600526835000, v:0}, last_push_gc_task_ts_:1727334198745636, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:59.920573] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=26] tx ctx memtable get rec scn(this={ObITable:{this:0x7f92a22a4080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f92a22a4080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:00:59.920586] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1723174350926847000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-09-26 16:00:59.920602] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:00:59.920626] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=22] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}, min_rec_scn={val:1726163586617842000, v:0}, ls_id_={id:1}) [2024-09-26 16:00:59.920638] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:59.920642] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=4] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:59.920644] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=1] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:00:59.920650] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1726163586617842000, v:0}, checkpoint_scn_in_ls_meta={val:1726163586617842000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-09-26 16:00:59.920658] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}, curr_base_lsn={lsn:627966418944}, new_base_lsn={lsn:627966418944}, lsn={lsn:627966418944}) [2024-09-26 16:00:59.920663] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=4] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:627966418944}, ls->get_ls_id()={id:1}) [2024-09-26 16:00:59.920666] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-09-26 16:00:59.929987] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337659846058]}) [2024-09-26 16:00:59.937895] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:00:59.937995] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=48, refresh_score_time=14, wash_time=1) [2024-09-26 16:00:59.938603] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=975167, clean_num=31457) [2024-09-26 16:00:59.943332] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:00:59.943342] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.973562] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:00:59.973570] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:00:59.978821] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [2708][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=166] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"2GB", log_disk_size:"2GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, mode:0, create_timestamp:1710828706488537, is_removed:false}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=15, log_id=1828900, offset=14449774}, ls_meta_entry:[10007](ver=0,mode=0,seq=609892), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:2}, create_status:1}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:49, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:80400898, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:1735, recv_lp_rpc_cnt:0, recv_mysql_cnt:1151, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:9765347, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:group_id = 2,queue_size = 0,recv_req_cnt = 7584,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334211521784 group_id = 10,queue_size = 0,recv_req_cnt = 3442,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334216415596 group_id = 9,queue_size = 0,recv_req_cnt = 7056,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334199387293 group_id = 1,queue_size = 0,recv_req_cnt = 4466,min_worker_cnt = 8,max_worker_cnt = 8,worker_cnt = 0,token_change = 1727336105580026 , rpc_stat_info: pcode=0x150a:cnt=20 pcode=0x150b:cnt=20 pcode=0x4a9:cnt=10 pcode=0x113:cnt=5, token_change_ts:1727334199363389, tenant_role:1}) [2024-09-26 16:00:59.979001] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [2708][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=179] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1727334198122806, is_removed:false}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:2}, create_status:1}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:12, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:2085440, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3254851, workers:12, nesting workers:0, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1727334198125585, tenant_role:0}) [2024-09-26 16:00:59.987580] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:00:59.987693] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=3] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-09-26 16:00:59.987701] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=6] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:59.987705] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=3] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:00:59.987708] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=3] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:59.989371] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] Dumping All Cache Objs(alloc_obj_list.count()=3, alloc_obj_list=[{obj_id:78, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157779}, {obj_id:79, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:5, added_to_lc:true, mem_used:175172}]) [2024-09-26 16:00:59.989383] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=11] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:00:59.991279] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:00:59.994484] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:00:59.994496] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337659994481) [2024-09-26 16:00:59.994501] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337659893843, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:00:59.994518] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223389229889, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:00:59.994524] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:00:59.994532] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:00:59.994537] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:00:59.994539] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:00:59.994545] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.994548] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:00:59.994549] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337659994504) [2024-09-26 16:01:00.004191] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:01:00.004200] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.026764] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185930-0-0] [lt=37][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:00.026775] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185930-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:00.026783] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185930-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:00.026788] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185930-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:00.030540] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337659846058]}) [2024-09-26 16:01:00.034324] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:00.034334] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.035618] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500837, support_ssl=false) [2024-09-26 16:01:00.035636] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221500837, sess.client_addr_="127.0.0.1:36380") [2024-09-26 16:01:00.035641] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f929f804030, session_id:3221500837, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:00.035681] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:00.035690] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:00.035727] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA5-0-0] [lt=8] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:00.035895] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA5-0-0] [lt=3][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:00.035900] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA5-0-0] [lt=4][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"rIX[AsB/c)<3)s-!\/pl"}, ret=-4043) [2024-09-26 16:01:00.035906] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA5-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:00.035909] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA5-0-0] [lt=2][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:00.035916] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA5-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:00.035937] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA5-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500837, proxy_sessid:0}) [2024-09-26 16:01:00.035941] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA5-0-0] [lt=3] mark session id unused(sessid=3221500837) [2024-09-26 16:01:00.035944] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA5-0-0] [lt=2][errcode=0] server close connection(sessid=3221500837, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:00.035947] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA5-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500837, proxy_sessid=0) [2024-09-26 16:01:00.035949] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA5-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:00.035951] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA5-0-0] [lt=1] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500837, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:00.035977] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f929f804030, session_id:3221500837, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337660035580, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:00.035984] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500837, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:00.035987] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f929f804030, session_id:3221500837, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337660035580, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:00.035991] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500837, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:00.061824] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:00.061837] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:01:00.061841] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:01:00.061844] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:01:00.061848] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:00.061850] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:00.061853] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:01:00.061855] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:00.061857] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:00.061859] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:00.061861] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:00.061863] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:00.061866] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:00.061868] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:00.061874] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:00.061876] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:00.061879] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:00.061881] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:00.061897] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=15][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:01:00.061901] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:00.061904] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:00.061910] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:00.061916] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:00.061918] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:00.061920] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:00.061926] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:00.061929] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85895-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:00.061933] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85895-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:01:00.061936] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:00.061938] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:00.061941] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85895-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:00.061944] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337660061754, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:00.061947] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85895-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:00.061951] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85895-0-0] [lt=3][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:01:00.061974] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85895-0-0] [lt=2][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:01:00.061979] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85895-0-0] [lt=4][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:01:00.064724] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:00.064754] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.068860] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:00.069132] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:00.069139] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:00.069142] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:00.069145] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:00.069148] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:00.069150] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:00.069153] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:00.069155] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:00.069157] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:00.069160] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:00.069162] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:00.069164] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:00.069166] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:00.069168] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:00.069172] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:00.069174] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:00.069177] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:00.069178] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:00.069180] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:01:00.069182] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:00.069185] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:00.069189] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:00.069195] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:00.069197] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:00.069198] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:00.069203] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:00.069206] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:00.069209] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D82-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:01:00.069211] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:00.069214] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:00.069215] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:00.069218] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337660069085, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:00.069220] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:00.069264] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:01:00.088171] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=16] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:00.094981] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:00.094990] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.095012] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:00.095016] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:00.095022] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337660095009) [2024-09-26 16:01:00.095026] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337659994504, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:00.095031] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:00.095040] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:00.095043] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:00.095324] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:00.095360] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=34][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:00.095365] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:00.095368] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:00.095372] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:00.095375] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:00.095378] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:00.095381] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:00.095383] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:00.095384] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:00.095387] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:00.095391] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:00.095393] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:00.095396] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:00.095410] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:00.095413] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:00.095416] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:00.095419] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:00.095421] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:00.095424] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:00.095427] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:00.095432] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:00.095437] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:00.095441] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:00.095443] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:00.095448] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:00.095451] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:00.095454] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885776-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:00.095457] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:00.095459] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:00.095461] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:00.095464] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337660095268, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:00.095468] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:00.095470] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:00.095510] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:00.095514] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337660095513, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=477, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:00.095520] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:00.095524] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:00.095541] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223389329659, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.095545] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:00.095552] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:00.095557] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:00.095559] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:00.095565] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.095567] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.095569] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337660095528) [2024-09-26 16:01:00.125292] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:00.125302] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.125311] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:00.130848] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337659846058]}) [2024-09-26 16:01:00.139069] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:00.139171] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=53, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:00.139828] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1006624, clean_num=31457) [2024-09-26 16:01:00.147771] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 10658.00) [2024-09-26 16:01:00.147781] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:00.147793] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:00.147797] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:00.155555] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:00.155562] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.185766] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:00.185793] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.191993] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=896496, replace_num=15728) [2024-09-26 16:01:00.192002] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:01:00.195196] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:00.195207] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337660195192) [2024-09-26 16:01:00.195213] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337660095029, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:00.195231] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223389431165, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.195240] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:00.195247] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:00.195252] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:00.195256] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:00.195262] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.195264] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.195266] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337660195215) [2024-09-26 16:01:00.201451] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:01:00.201458] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=5] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:01:00.201466] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:01:00.216129] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:00.216135] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.232751] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337659846058]}) [2024-09-26 16:01:00.232767] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D98A-0-0] [lt=6][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337660232722}) [2024-09-26 16:01:00.246482] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:00.246501] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.252238] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337659251211]) [2024-09-26 16:01:00.252251] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337659251211], now0=[mts=1727337659251211], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:00.252260] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337659251128, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:00.252296] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=34][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:01:00.252307] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337659251128, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:00.252318] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337659251128, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:00.252340] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=21][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:00.252347] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:00.252351] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:00.252354] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:00.252359] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:01:00.252364] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=22, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:00.252370] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5] will sleep(sleep_us=22000, remain_us=6741962, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:01:00.256518] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:00.256529] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:00.256534] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:00.256537] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:00.256541] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:00.256543] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:00.256546] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:00.256549] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:00.256550] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:00.256552] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:00.256555] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:00.256557] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:00.256559] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:00.256561] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:00.256567] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:00.256569] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:00.256572] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:00.256574] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:00.256577] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:01:00.256579] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:00.256581] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:00.256586] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:00.256591] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:00.256593] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:00.256594] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:00.256600] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:00.256603] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:00.256605] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:01:00.256608] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A4-0-0] [lt=3][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.274749] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:22, local_retry_times:22, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:00.274764] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:00.274768] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:00.274769] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:00.274774] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:00.274805] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:01:00.274809] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:00.277683] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:00.277689] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.295242] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:00.295259] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337660295239) [2024-09-26 16:01:00.295264] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337660195215, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:00.295282] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223389530965, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.295287] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:00.295294] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:00.295303] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:00.295306] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:00.295311] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.295313] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.295315] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337660295267) [2024-09-26 16:01:00.307731] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:00.307741] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.318712] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=21][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337660245803, now=[mts=1727337659318720]) [2024-09-26 16:01:00.318724] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337660245803, now=[mts=1727337659318720], now0=[mts=1727337659318720], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:00.318731] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926901c400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337659318556, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:00.318747] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926901c400}) [2024-09-26 16:01:00.318755] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926901c400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337659318556, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:00.318762] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926901c400, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337659318556, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337658316043, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926901c400}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:00.318778] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:00.318782] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:00.318786] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:00.318791] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=4][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:00.318793] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:00.318797] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:00.318803] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:01:00.318809] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:00.318812] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:00.318813] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:01:00.318819] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:01:00.318822] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2002779) [2024-09-26 16:01:00.318838] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=15][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:01:00.318842] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:00.318845] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:01:00.318847] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:01:00.318850] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:01:00.318855] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:00.318886] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=4][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:01:00.318890] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:01:00.318894] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=4][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:01:00.318897] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:00.318901] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:00.318903] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:01:00.319012] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE1-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:01:00.319021] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:01:00.319023] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:01:00.319038] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:00.319042] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:01:00.319308] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:01:00.319316] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=8][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:00.333358] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337659846058]}) [2024-09-26 16:01:00.338209] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:00.338230] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.340312] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:00.340414] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:00.341016] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1038081, clean_num=31457) [2024-09-26 16:01:00.347136] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.347146] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337660347134], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:00.347153] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337660347134], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:00.368504] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:00.368513] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.390948] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=5] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:00.391973] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=15] current monitor number(seq_=-1) [2024-09-26 16:01:00.392023] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:01:00.395858] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:00.395870] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337660395855) [2024-09-26 16:01:00.395886] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337660295267, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:00.395905] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223389631389, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.395914] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:00.395921] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:00.395926] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:00.395930] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:00.395936] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.395938] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.395940] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337660395890) [2024-09-26 16:01:00.398814] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:00.398820] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.426155] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D4214-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:00.429046] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:00.429051] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.434962] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337660347134]}) [2024-09-26 16:01:00.458394] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1727337660458393, bytes: 14705620, bw: 0.003349 MB/s, add_ts: 1008008, add_bytes: 3540 [2024-09-26 16:01:00.459344] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:00.459359] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.465217] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.465216] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.465205] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.465223] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.465223] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.465228] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.465228] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.465227] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.465232] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.466167] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.466173] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.466177] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.466179] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.466182] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.466184] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.466200] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.466206] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.466211] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.467138] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.467144] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.467147] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.467159] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.467170] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.467171] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.467172] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.467174] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.467174] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.468132] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.468135] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.468137] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.468138] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.468140] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.468140] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.468141] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.468144] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.468146] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.469124] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.469125] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.469129] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.469132] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.469134] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.469136] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.469137] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.469139] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.469132] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.470053] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.470059] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.470061] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:00.470097] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.470105] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.471036] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.471044] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.471107] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.472020] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.472079] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.472083] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.472985] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.472994] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.472995] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.474967] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.474975] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.474978] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.475922] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.475937] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.475939] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.476942] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.476946] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.476956] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.477872] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.477881] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.477880] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.478853] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.478859] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.479874] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.479885] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.479887] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.480806] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.480814] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.480830] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.481790] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.481799] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.481864] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.482323] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.482330] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.482416] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.482989] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.482989] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.482996] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.483779] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.483787] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.483795] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.484713] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.484722] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.484721] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.485703] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.485713] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.485733] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.486678] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.486684] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.486699] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.487650] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.487659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.487662] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.488617] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.489603] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.489631] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:00.489654] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.489659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.489677] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.490598] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.490615] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.490626] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.491575] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.491578] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.491579] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.492547] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.492552] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.492561] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.493489] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.493499] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.493500] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.494489] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.494490] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.494497] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.495473] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.496460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.496462] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.496468] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.496489] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:00.496500] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337660496486) [2024-09-26 16:01:00.496505] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337660395890, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:00.496524] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223389732957, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.496530] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:00.496538] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:00.496543] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:00.496557] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:00.496564] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.496569] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.496571] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337660496508) [2024-09-26 16:01:00.497437] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.497446] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.497448] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.498404] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.498416] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.498435] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:00.519865] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:00.519871] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.522972] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185931-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:00.522979] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185931-0-0] [lt=7][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:00.522988] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185931-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:00.522995] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185931-0-0] [lt=6][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:00.535558] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337660347134]}) [2024-09-26 16:01:00.541520] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:00.541620] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=48, refresh_score_time=14, wash_time=1) [2024-09-26 16:01:00.542223] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1069538, clean_num=31457) [2024-09-26 16:01:00.547269] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6616, mem_used=12478464) [2024-09-26 16:01:00.550148] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:01:00.550156] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.565788] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=9] gc stale ls task succ [2024-09-26 16:01:00.581431] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:00.581442] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.592110] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:01:00.596999] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:00.597016] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337660596994) [2024-09-26 16:01:00.597022] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337660496508, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:00.597029] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:00.597041] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:00.597044] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:00.597423] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885777-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:00.597434] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885777-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.597483] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885777-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:00.597519] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337660597518, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=483, wlock_time=11, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:00.597529] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:00.597547] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223389832652, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.597552] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:00.597558] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:00.597562] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:00.597565] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:00.597570] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.597572] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.597574] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337660597533) [2024-09-26 16:01:00.601536] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.601543] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.601749] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.601761] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.612678] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:00.612690] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.633198] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] ====== traversal_flush timer task ====== [2024-09-26 16:01:00.633239] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d76030, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6163825430799493735, dag_cnt=1, dag_type_cnts=1) [2024-09-26 16:01:00.633248] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] success to create and add dag(ret=0, dag=0x7f9264d76030) [2024-09-26 16:01:00.633250] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1961) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(dag_cnt=1, map_size=1) [2024-09-26 16:01:00.633257] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1971) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(running_dag_net_map_size=0, blocking_dag_net_list_size=0) [2024-09-26 16:01:00.633260] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(priority="PRIO_COMPACTION_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=1, waiting_dag_count=0) [2024-09-26 16:01:00.633266] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(priority="PRIO_HA_HIGH", low_limit=8, up_limit=8, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:00.633268] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_COMPACTION_MID", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:00.633251] INFO [STORAGE] flush (ob_tx_data_memtable.cpp:917) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] [TX DATA MERGE]schedule flush tx data memtable task done(ret=0, ret="OB_SUCCESS", get_ls_id()={id:1}, this=0x7f92a2294080, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:4, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337660633251, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:01:00.633272] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(priority="PRIO_HA_MID", low_limit=5, up_limit=5, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:00.633275] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(priority="PRIO_COMPACTION_LOW", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:00.633276] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(priority="PRIO_HA_LOW", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:00.633278] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_DDL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:00.633279] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(priority="PRIO_DDL_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:00.633281] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_TTL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:00.633282] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:01:00.633286] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=182) [2024-09-26 16:01:00.633288] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:01:00.633290] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:01:00.633292] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:01:00.633293] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:01:00.633297] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=1) [2024-09-26 16:01:00.633300] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=2) [2024-09-26 16:01:00.633301] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:01:00.633303] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:01:00.633305] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:01:00.633289] INFO [MDS] flush (mds_table_mgr.cpp:172) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=20] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:01:00.633309] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:01:00.633311] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:01:00.633313] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:01:00.633314] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:01:00.633316] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:01:00.633317] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:01:00.633319] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:01:00.633320] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:01:00.633321] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:01:00.633324] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=21] add dag success(dag=0x7f9264d76560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6659188001345271846, dag_cnt=2, dag_type_cnts=1) [2024-09-26 16:01:00.633325] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633326] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d76560) [2024-09-26 16:01:00.633328] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633330] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633331] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633333] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633334] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633336] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633330] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, create_flush_dag_time:1727337660633330, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452867, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200240}, checkpoint_unit_ptr:0x7f92a1a70850}, is_freeze_clock:true}) [2024-09-26 16:01:00.633339] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633341] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633342] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633345] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633347] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633349] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633339] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337660633318, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633361] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633363] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633364] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633366] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633366] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=23] add dag success(dag=0x7f9264d76a90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-9069613569758001279, dag_cnt=3, dag_type_cnts=2) [2024-09-26 16:01:00.633368] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633369] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d76a90) [2024-09-26 16:01:00.633370] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633371] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633371] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, create_flush_dag_time:1727337660633371, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452892, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200489}, checkpoint_unit_ptr:0x7f92a1a70080}, is_freeze_clock:true}) [2024-09-26 16:01:00.633373] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633375] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633378] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633380] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633375] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337660633362, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633382] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633383] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633385] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633385] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d76fc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1288986994123435108, dag_cnt=4, dag_type_cnts=3) [2024-09-26 16:01:00.633386] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633387] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d76fc0) [2024-09-26 16:01:00.633388] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:00.633389] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633388] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337660633388, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452911, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847222}, checkpoint_unit_ptr:0x7f92a1a71020}, is_freeze_clock:true}) [2024-09-26 16:01:00.633391] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-09-26 16:01:00.633392] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-09-26 16:01:00.633394] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-09-26 16:01:00.633396] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-09-26 16:01:00.633392] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337660633383, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633398] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:00.633399] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:00.633400] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d774f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5355160120942094702, dag_cnt=5, dag_type_cnts=4) [2024-09-26 16:01:00.633401] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:00.633402] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d774f0) [2024-09-26 16:01:00.633403] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:00.633404] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:00.633403] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337660633403, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452929, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847223}, checkpoint_unit_ptr:0x7f92a1a717f0}, is_freeze_clock:true}) [2024-09-26 16:01:00.633406] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:00.633408] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:00.633411] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:00.633413] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:00.633406] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337660633398, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633414] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:00.633416] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:00.633417] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:00.633428] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:00.633430] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:00.633432] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:00.633418] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405042) [2024-09-26 16:01:00.633434] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:00.633435] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:00.633437] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:00.633438] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:00.633439] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:01:00.633440] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633442] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:00.633446] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633448] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:00.633462] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633444] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405070) [2024-09-26 16:01:00.633463] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:00.633465] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633465] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:01:00.633466] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:00.633468] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633472] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:00.633474] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633470] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405087) [2024-09-26 16:01:00.633475] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:00.633477] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633477] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:01:00.633478] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:00.633480] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633481] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:00.633483] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633484] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:00.633480] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405104) [2024-09-26 16:01:00.633486] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:01:00.633487] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:00.633488] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0) [2024-09-26 16:01:00.633490] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, scheduled_task_count=0) [2024-09-26 16:01:00.633491] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] add dag success(dag=0x7f9264d77a20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1994541446894784994, dag_cnt=6, dag_type_cnts=5) [2024-09-26 16:01:00.633491] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0) [2024-09-26 16:01:00.633493] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, scheduled_task_count=0) [2024-09-26 16:01:00.633493] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d77a20) [2024-09-26 16:01:00.633495] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0) [2024-09-26 16:01:00.633496] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-09-26 16:01:00.633495] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, create_flush_dag_time:1727337660633495, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209556884, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:226}, checkpoint_unit_ptr:0x7f92a1a73f00}, is_freeze_clock:true}) [2024-09-26 16:01:00.633498] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0) [2024-09-26 16:01:00.633500] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-09-26 16:01:00.633503] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0) [2024-09-26 16:01:00.633499] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337660633488, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633505] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, scheduled_task_count=0) [2024-09-26 16:01:00.633507] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_net_count=0) [2024-09-26 16:01:00.633509] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_net_count=0) [2024-09-26 16:01:00.633510] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_net_count=0) [2024-09-26 16:01:00.633511] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_net_count=0) [2024-09-26 16:01:00.633512] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d77f50, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8791616776033068452, dag_cnt=7, dag_type_cnts=6) [2024-09-26 16:01:00.633512] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_net_count=0) [2024-09-26 16:01:00.633513] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d77f50) [2024-09-26 16:01:00.633513] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_net_count=0) [2024-09-26 16:01:00.633515] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_net_count=0) [2024-09-26 16:01:00.633516] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_net_count=0) [2024-09-26 16:01:00.633515] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337660633515, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405141, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847408}, checkpoint_unit_ptr:0x7f92a1a746d0}, is_freeze_clock:true}) [2024-09-26 16:01:00.633519] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1991) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=184) [2024-09-26 16:01:00.633518] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337660633506, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633527] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337660633523, task_id:YB42C0A8002D-0006230056D8C6FC-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.633534] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=5] schedule one task(task=0x7f9264d763f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.633543] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337660633541, task_id:YB42C0A8002D-0006230056D8C6FD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.633557] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=13] schedule one task(task=0x7f9264d76920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.633567] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337660633566, task_id:YB42C0A8002D-0006230056D8C6FE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.633569] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=2] schedule one task(task=0x7f9264d76e50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.633574] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660633573, task_id:YB42C0A8002D-0006230056D8C6FF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.633577] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=3] schedule one task(task=0x7f9264d77380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.633582] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660633581, task_id:YB42C0A8002D-0006230056D8C700-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.633585] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=3] schedule one task(task=0x7f9264d778b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.633572] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=8][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.633588] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660633588, task_id:YB42C0A8002D-0006230056D8C701-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.633588] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=16][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.633591] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=2] schedule one task(task=0x7f9264d77de0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.633592] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726157244792838000) [2024-09-26 16:01:00.633603] INFO [STORAGE] ready_for_flush (ob_tx_data_memtable.cpp:865) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=10] memtable is frozen yet.(this=0x7f92a2294080) [2024-09-26 16:01:00.633610] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.633615] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.633617] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:01:00.633616] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.633620] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.633622] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:01:00.633628] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.633624] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.633638] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=11] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847222}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:01:00.633642] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.633645] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.633647] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:01:00.633651] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.633651] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.633651] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.633653] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.633655] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:01:00.633658] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.633662] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:01:00.633670] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.633671] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.633674] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.633675] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.633678] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.633681] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:01:00.633689] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241777443612000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.633739] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d78480, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6550230011160095889, dag_cnt=8, dag_type_cnts=7) [2024-09-26 16:01:00.633743] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] success to create and add dag(ret=0, dag=0x7f9264d78480) [2024-09-26 16:01:00.633745] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337660633745, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405158, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847409}, checkpoint_unit_ptr:0x7f92a1a75670}, is_freeze_clock:true}) [2024-09-26 16:01:00.633751] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337660633525, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633767] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=10] add dag success(dag=0x7f9264d789b0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6809498982591245614, dag_cnt=9, dag_type_cnts=8) [2024-09-26 16:01:00.633769] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d789b0) [2024-09-26 16:01:00.633771] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, create_flush_dag_time:1727337660633771, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405174, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:200004}, checkpoint_unit_ptr:0x7f92a1a74ea0}, is_freeze_clock:true}) [2024-09-26 16:01:00.633774] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337660633761, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633784] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d78ee0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6815944285528637599, dag_cnt=10, dag_type_cnts=9) [2024-09-26 16:01:00.633786] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d78ee0) [2024-09-26 16:01:00.633788] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, create_flush_dag_time:1727337660633788, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209506214, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1}, checkpoint_unit_ptr:0x7f92a1a76610}, is_freeze_clock:true}) [2024-09-26 16:01:00.633791] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337660633782, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633800] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d79410, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-4182959058374342994, dag_cnt=11, dag_type_cnts=10) [2024-09-26 16:01:00.633802] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d79410) [2024-09-26 16:01:00.633804] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, create_flush_dag_time:1727337660633804, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405204, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:412}, checkpoint_unit_ptr:0x7f92a1a75e40}, is_freeze_clock:true}) [2024-09-26 16:01:00.633808] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337660633798, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633817] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d79940, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1586518267935340, dag_cnt=12, dag_type_cnts=11) [2024-09-26 16:01:00.633819] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d79940) [2024-09-26 16:01:00.633821] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, create_flush_dag_time:1727337660633821, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405220, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:444}, checkpoint_unit_ptr:0x7f92a1a775b0}, is_freeze_clock:true}) [2024-09-26 16:01:00.633825] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337660633814, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633834] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d79e70, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3044367941103776476, dag_cnt=13, dag_type_cnts=12) [2024-09-26 16:01:00.633836] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d79e70) [2024-09-26 16:01:00.633837] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, create_flush_dag_time:1727337660633837, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455964, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:309}, checkpoint_unit_ptr:0x7f92a1a76de0}, is_freeze_clock:true}) [2024-09-26 16:01:00.633841] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337660633831, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633849] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7a3a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1596830851104282047, dag_cnt=14, dag_type_cnts=13) [2024-09-26 16:01:00.633852] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7a3a0) [2024-09-26 16:01:00.633853] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, create_flush_dag_time:1727337660633853, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455986, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:331}, checkpoint_unit_ptr:0x7f92a1a77d80}, is_freeze_clock:true}) [2024-09-26 16:01:00.633857] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337660633847, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633865] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7a8d0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2360195962081676673, dag_cnt=15, dag_type_cnts=14) [2024-09-26 16:01:00.633867] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7a8d0) [2024-09-26 16:01:00.633871] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, create_flush_dag_time:1727337660633871, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405262, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:458}, checkpoint_unit_ptr:0x7f92a1a7a490}, is_freeze_clock:true}) [2024-09-26 16:01:00.633875] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337660633863, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633885] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7ae00, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5675615614797020460, dag_cnt=16, dag_type_cnts=15) [2024-09-26 16:01:00.633887] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7ae00) [2024-09-26 16:01:00.633889] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, create_flush_dag_time:1727337660633889, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405279, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:487}, checkpoint_unit_ptr:0x7f92a1a79cc0}, is_freeze_clock:true}) [2024-09-26 16:01:00.633892] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337660633882, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633902] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7b330, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-612435342908088006, dag_cnt=17, dag_type_cnts=16) [2024-09-26 16:01:00.633904] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7b330) [2024-09-26 16:01:00.633905] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, create_flush_dag_time:1727337660633905, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405296, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:455}, checkpoint_unit_ptr:0x7f92a1a794f0}, is_freeze_clock:true}) [2024-09-26 16:01:00.633909] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337660633899, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633917] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7b860, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7868818205990873505, dag_cnt=18, dag_type_cnts=17) [2024-09-26 16:01:00.633919] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7b860) [2024-09-26 16:01:00.633921] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, create_flush_dag_time:1727337660633921, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405313, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:492}, checkpoint_unit_ptr:0x7f92a1a78550}, is_freeze_clock:true}) [2024-09-26 16:01:00.633924] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337660633915, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633933] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7bd90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7071731214611448811, dag_cnt=19, dag_type_cnts=18) [2024-09-26 16:01:00.633935] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7bd90) [2024-09-26 16:01:00.633937] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, create_flush_dag_time:1727337660633937, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405347, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:501}, checkpoint_unit_ptr:0x7f92a1a78d20}, is_freeze_clock:true}) [2024-09-26 16:01:00.633940] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337660633931, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633949] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7c2c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8731750006000903454, dag_cnt=20, dag_type_cnts=19) [2024-09-26 16:01:00.633951] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7c2c0) [2024-09-26 16:01:00.633953] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, create_flush_dag_time:1727337660633953, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405365, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:472}, checkpoint_unit_ptr:0x7f92a1a7ac60}, is_freeze_clock:true}) [2024-09-26 16:01:00.633956] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337660633947, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633965] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7c7f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6766388540859538791, dag_cnt=21, dag_type_cnts=20) [2024-09-26 16:01:00.633967] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7c7f0) [2024-09-26 16:01:00.633969] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, create_flush_dag_time:1727337660633969, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405384, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:342}, checkpoint_unit_ptr:0x7f92a1a7cba0}, is_freeze_clock:true}) [2024-09-26 16:01:00.633972] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337660633963, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633981] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7cd20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2986199386776543176, dag_cnt=22, dag_type_cnts=21) [2024-09-26 16:01:00.633983] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7cd20) [2024-09-26 16:01:00.633985] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337660633984, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405418, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:140}, checkpoint_unit_ptr:0x7f92a1a7b430}, is_freeze_clock:true}) [2024-09-26 16:01:00.633988] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337660633979, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.633997] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7d250, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6947063304478012270, dag_cnt=23, dag_type_cnts=22) [2024-09-26 16:01:00.633999] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7d250) [2024-09-26 16:01:00.634000] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337660634000, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405436, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101009}, checkpoint_unit_ptr:0x7f92a1a7bc00}, is_freeze_clock:true}) [2024-09-26 16:01:00.634004] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337660633995, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634013] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7d780, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5843378789243662202, dag_cnt=24, dag_type_cnts=23) [2024-09-26 16:01:00.634015] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7d780) [2024-09-26 16:01:00.634016] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337660634016, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405453, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101010}, checkpoint_unit_ptr:0x7f92a1a7c3d0}, is_freeze_clock:true}) [2024-09-26 16:01:00.634020] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337660634010, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634028] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7dcb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1841823684513746176, dag_cnt=25, dag_type_cnts=24) [2024-09-26 16:01:00.634030] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7dcb0) [2024-09-26 16:01:00.634032] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, create_flush_dag_time:1727337660634032, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405470, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:323}, checkpoint_unit_ptr:0x7f92a1a7d370}, is_freeze_clock:true}) [2024-09-26 16:01:00.634035] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337660634026, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634044] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7e1e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-2678407261265885087, dag_cnt=26, dag_type_cnts=25) [2024-09-26 16:01:00.634046] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7e1e0) [2024-09-26 16:01:00.634048] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, create_flush_dag_time:1727337660634048, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405487, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:377}, checkpoint_unit_ptr:0x7f92a1a7e310}, is_freeze_clock:true}) [2024-09-26 16:01:00.634051] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337660634042, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634062] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d7e710, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7175089551615529855, dag_cnt=27, dag_type_cnts=26) [2024-09-26 16:01:00.634063] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d7e710) [2024-09-26 16:01:00.634065] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, create_flush_dag_time:1727337660634065, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405503, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:297}, checkpoint_unit_ptr:0x7f92a1a7eae0}, is_freeze_clock:true}) [2024-09-26 16:01:00.634068] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337660634059, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634077] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7ec40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3730526293242960203, dag_cnt=28, dag_type_cnts=27) [2024-09-26 16:01:00.634079] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7ec40) [2024-09-26 16:01:00.634081] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337660634081, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405519, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101065}, checkpoint_unit_ptr:0x7f92a1a7db40}, is_freeze_clock:true}) [2024-09-26 16:01:00.634084] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337660634075, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634103] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=16] add dag success(dag=0x7f9264d7f170, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=4663792628280492528, dag_cnt=29, dag_type_cnts=28) [2024-09-26 16:01:00.634106] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d7f170) [2024-09-26 16:01:00.634108] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337660634108, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405536, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:333}, checkpoint_unit_ptr:0x7f92a3bc4850}, is_freeze_clock:true}) [2024-09-26 16:01:00.634111] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337660634100, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634120] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7f6a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8399557287176188414, dag_cnt=30, dag_type_cnts=29) [2024-09-26 16:01:00.634123] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d7f6a0) [2024-09-26 16:01:00.634124] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337660634124, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405552, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101066}, checkpoint_unit_ptr:0x7f92a3bc5020}, is_freeze_clock:true}) [2024-09-26 16:01:00.634128] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337660634118, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634136] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7fbd0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8068938909778324349, dag_cnt=31, dag_type_cnts=30) [2024-09-26 16:01:00.634139] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d7fbd0) [2024-09-26 16:01:00.634140] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, create_flush_dag_time:1727337660634140, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405569, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:334}, checkpoint_unit_ptr:0x7f92a3bc57f0}, is_freeze_clock:true}) [2024-09-26 16:01:00.634144] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337660634134, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634153] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d80100, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7665372330263152188, dag_cnt=32, dag_type_cnts=31) [2024-09-26 16:01:00.634156] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d80100) [2024-09-26 16:01:00.634158] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, create_flush_dag_time:1727337660634158, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405586, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:451}, checkpoint_unit_ptr:0x7f92a3bc4080}, is_freeze_clock:true}) [2024-09-26 16:01:00.634161] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337660634150, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634171] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d80630, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6759548225246904317, dag_cnt=33, dag_type_cnts=32) [2024-09-26 16:01:00.634174] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d80630) [2024-09-26 16:01:00.634176] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, create_flush_dag_time:1727337660634175, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405602, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:378}, checkpoint_unit_ptr:0x7f92a1a7f2b0}, is_freeze_clock:true}) [2024-09-26 16:01:00.634179] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337660634169, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634198] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=14] add dag success(dag=0x7f9264d80b60, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8944614441376604790, dag_cnt=34, dag_type_cnts=33) [2024-09-26 16:01:00.634201] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d80b60) [2024-09-26 16:01:00.634206] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, create_flush_dag_time:1727337660634206, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209456003, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:121}, checkpoint_unit_ptr:0x7f92a3bc5fc0}, is_freeze_clock:true}) [2024-09-26 16:01:00.634210] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337660634194, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634220] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d81090, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=507863347806783454, dag_cnt=35, dag_type_cnts=34) [2024-09-26 16:01:00.634223] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d81090) [2024-09-26 16:01:00.634225] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, create_flush_dag_time:1727337660634225, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405633, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:215}, checkpoint_unit_ptr:0x7f92a3bc86d0}, is_freeze_clock:true}) [2024-09-26 16:01:00.634228] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337660634217, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634238] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d815c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1604282355669434429, dag_cnt=36, dag_type_cnts=35) [2024-09-26 16:01:00.634240] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d815c0) [2024-09-26 16:01:00.634242] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, create_flush_dag_time:1727337660634242, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405649, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:456}, checkpoint_unit_ptr:0x7f92a3bc6790}, is_freeze_clock:true}) [2024-09-26 16:01:00.634246] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337660634235, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634255] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d81af0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3553649416163308074, dag_cnt=37, dag_type_cnts=36) [2024-09-26 16:01:00.634258] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d81af0) [2024-09-26 16:01:00.634262] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337660634262, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405666, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:154}, checkpoint_unit_ptr:0x7f92a3bc6f60}, is_freeze_clock:true}) [2024-09-26 16:01:00.634266] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337660634252, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634275] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d82020, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8097406237500329300, dag_cnt=38, dag_type_cnts=37) [2024-09-26 16:01:00.634278] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d82020) [2024-09-26 16:01:00.634279] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337660634279, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405682, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101015}, checkpoint_unit_ptr:0x7f92a3bc7730}, is_freeze_clock:true}) [2024-09-26 16:01:00.634283] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337660634272, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634294] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d82550, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3976350754940852513, dag_cnt=39, dag_type_cnts=38) [2024-09-26 16:01:00.634297] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d82550) [2024-09-26 16:01:00.634299] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337660634299, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405699, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101016}, checkpoint_unit_ptr:0x7f92a3bc7f00}, is_freeze_clock:true}) [2024-09-26 16:01:00.634302] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337660634291, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634311] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d82a80, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8720304653345806796, dag_cnt=40, dag_type_cnts=39) [2024-09-26 16:01:00.634314] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d82a80) [2024-09-26 16:01:00.634316] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, create_flush_dag_time:1727337660634316, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405716, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101063}, checkpoint_unit_ptr:0x7f92a3bca610}, is_freeze_clock:true}) [2024-09-26 16:01:00.634321] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337660634308, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634334] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d82fb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1830562711513853292, dag_cnt=41, dag_type_cnts=40) [2024-09-26 16:01:00.634337] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d82fb0) [2024-09-26 16:01:00.634339] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, create_flush_dag_time:1727337660634338, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405733, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:452}, checkpoint_unit_ptr:0x7f92a3bc9e40}, is_freeze_clock:true}) [2024-09-26 16:01:00.634342] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337660634328, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634351] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d83560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1536604874905918360, dag_cnt=42, dag_type_cnts=41) [2024-09-26 16:01:00.634354] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d83560) [2024-09-26 16:01:00.634356] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337660634356, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405750, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:328}, checkpoint_unit_ptr:0x7f92a3bc8ea0}, is_freeze_clock:true}) [2024-09-26 16:01:00.634359] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337660634349, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634369] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d83a90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6933628521298738662, dag_cnt=43, dag_type_cnts=42) [2024-09-26 16:01:00.634371] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d83a90) [2024-09-26 16:01:00.634373] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337660634373, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405767, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:330}, checkpoint_unit_ptr:0x7f92a3bc9670}, is_freeze_clock:true}) [2024-09-26 16:01:00.634376] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337660634366, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634386] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d83fc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1056199536334423564, dag_cnt=44, dag_type_cnts=43) [2024-09-26 16:01:00.634388] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d83fc0) [2024-09-26 16:01:00.634392] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337660634392, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405784, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:329}, checkpoint_unit_ptr:0x7f92a3bcb5b0}, is_freeze_clock:true}) [2024-09-26 16:01:00.634396] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337660634383, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634406] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d844f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7405463607918801432, dag_cnt=45, dag_type_cnts=44) [2024-09-26 16:01:00.634409] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d844f0) [2024-09-26 16:01:00.634411] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, create_flush_dag_time:1727337660634411, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405801, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:332}, checkpoint_unit_ptr:0x7f92a3bcade0}, is_freeze_clock:true}) [2024-09-26 16:01:00.634414] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337660634403, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634423] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d84a20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6699723950958743903, dag_cnt=46, dag_type_cnts=45) [2024-09-26 16:01:00.634426] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d84a20) [2024-09-26 16:01:00.634430] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, create_flush_dag_time:1727337660634430, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405818, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101067}, checkpoint_unit_ptr:0x7f92a3bcbd80}, is_freeze_clock:true}) [2024-09-26 16:01:00.634434] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337660634421, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.634441] INFO [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:655) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] traversal_flush successfully(ls_->get_ls_id()={id:1}, flush_tasks=[{table:0x7f92a1a70850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a70080, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71020, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a717f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71fc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72790, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72f60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73730, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73f00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a746d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75670, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a74ea0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76610, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75e40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a775b0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76de0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a77d80, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7a490, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a79cc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a794f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78550, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78d20, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7ac60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7cba0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7b430, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7bc00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7c3d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7d370, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7e310, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7eae0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7db40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a3bc4850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}]) [2024-09-26 16:01:00.634507] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=63] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-09-26 16:01:00.636143] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337660347134]}) [2024-09-26 16:01:00.636154] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855F3-0-0] [lt=9] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:00.638118] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:47, tid:2932}]) [2024-09-26 16:01:00.642951] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21] success to notify_need_writing_throttling True [2024-09-26 16:01:00.642967] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.664471] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:00.665428] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=8][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337660347134]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:01:00.665468] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=39][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:01:00.673217] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:00.673224] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.689840] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:00.697647] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:00.697660] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337660697632) [2024-09-26 16:01:00.697666] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337660597025, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:00.697702] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223389934079, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.697717] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:00.697725] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:00.697730] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:00.697733] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:00.697741] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.697743] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.697745] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337660697670) [2024-09-26 16:01:00.697768] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.697788] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=19] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337660633383, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:01:00.697782] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.697801] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=19] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337660633398, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:01:00.697796] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.697810] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.697791] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726163995122514000, tablet={this:0x7f92b862bfe0, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:505, tablet_addr:[816017-1849-0-2305843645944356864-2](offset=1761280,size=593,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:200489}, data_tablet_id:{id:200489}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725991206240736000, v:0}, clog_checkpoint_scn:{val:1726163995200406000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726163995122514000, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1725991205062061000, cur_report_version:1725991205062061000, data_checksum:1210484554, row_count:46137450}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1725120000848336, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726201924319685000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1725120000848336}, table_store_addr:{ptr:{this:0x7f92b862c674, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:4, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305877511761428480-0](offset=1703936,size=32133,type=2,seq=0), type:"MAJOR", tablet_id:{id:200489}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1725991205062061000, v:0}}, snapshot_version:1725991205062061000, max_merge_version:1725991203769806000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305849822107303301-0](offset=1736069,size=6345,type=2,seq=0), type:"MINOR", tablet_id:{id:200489}, scn_range:{start_scn:{val:1725991206240736000, v:0}, end_scn:{val:1726076389100953000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726076389025660000, upper_trans_version:1726076389025660000}, {i:1, addr:[816017-1849-0-2305844368572585550-0](offset=1742414,size=1266,type=2,seq=0), type:"MINI", tablet_id:{id:200489}, scn_range:{start_scn:{val:1726076389100953000, v:0}, end_scn:{val:1726077259218062001, v:0}}, contain_uncommitted_row:"yes", max_merge_version:1726077257041685000, upper_trans_version:1726077259766338000}, {i:2, addr:[816017-1849-0-2305843929412180800-0](offset=1743680,size=857,type=2,seq=0), type:"MINI", tablet_id:{id:200489}, scn_range:{start_scn:{val:1726077259218062001, v:0}, end_scn:{val:1726077600564631000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726077598368094000, upper_trans_version:1726077598368094000}, {i:3, addr:[816017-1849-0-2305849899416723097-0](offset=1744537,size=6417,type=2,seq=0), type:"MINOR", tablet_id:{id:200489}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726163995200406000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726163995200406000, upper_trans_version:1726163995200406000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337660633362, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843710368858112-2](offset=1753088,size=653,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843165981754119-0](offset=1753863,size=146,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255102325657-0](offset=1754009,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1725991205062061000}, medium_info_list:{ptr:(nil), addr:[816017-1849-0-2305843901494906880-0](offset=1757184,size=831,type=2,seq=0)}, auto_inc_seq:{ptr:0x7f92b862cafc, auto_inc_seq:{version:1, intervals_count:1, intervals:{tablet_id:{id:0}, start:55640001, end:9223372036854775807}}, addr:[816017-1849-0-2305843140211950221-0](offset=1753741,size=122,type=2,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1725120001569133000, v:0}, create_commit_version:1725120001569133000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a70080, 0x7f92a1a73730, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.697869] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=77] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:01:00.697882] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=6] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.697907] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=14] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:01:00.697915] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=6] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.697927] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:01:00.697926] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847222}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:01:00.697932] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.697952] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=15] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847223}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:01:00.697955] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:1152921504606847222}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847222}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660633605, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=64.31ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.697985] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=26] succeed to build merge ctx(tablet_id={id:1152921504606847223}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847223}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660633648, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=64.30ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.698219] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=40] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:01:00.698274] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=23] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:01:00.698944] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=17] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=8366, total_rows=58562) [2024-09-26 16:01:00.699074] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=15] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=7999, total_rows=55993) [2024-09-26 16:01:00.699358] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=6] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FF-0-0, dag_ret:0, dag_status:2, start_time:1727337660633577, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a7470, output_block_cnt_arr:0x7f92a23a7478, concurrent_cnt:1, estimate_row_cnt:329211, estimate_occupy_size:150994944, latest_update_ts:1727337660633605, estimated_finish_time:1727337677699430}) [2024-09-26 16:01:00.699373] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=15] succeed to init merge ctx(task={this:0x7f9264d77380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FF-0-0, dag_ret:0, dag_status:2, start_time:1727337660633577, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.699381] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=7] task finish process(ret=0, start_time=1727337660633603, end_time=1727337660699381, runtime=65778, *this={this:0x7f9264d77380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FF-0-0, dag_ret:0, dag_status:2, start_time:1727337660633577, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.699425] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=1] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.699517] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=7] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92a9ed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a64060, count:4, allocator:0x7f92c1d9e7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a64350, local_datums:0x7f9264a64350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:00.699557] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=35] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92a9ed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a68060, count:4, allocator:0x7f92c1d9f558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a68350, local_datums:0x7f9264a68350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:00.699637] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=6] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C700-0-0, dag_ret:0, dag_status:2, start_time:1727337660633585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad470, output_block_cnt_arr:0x7f92684ad478, concurrent_cnt:1, estimate_row_cnt:332829, estimate_occupy_size:150994944, latest_update_ts:1727337660633648, estimated_finish_time:1727337677699708}) [2024-09-26 16:01:00.699650] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f9264d778b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C700-0-0, dag_ret:0, dag_status:2, start_time:1727337660633585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.699667] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=16] task finish process(ret=0, start_time=1727337660633646, end_time=1727337660699666, runtime=66020, *this={this:0x7f9264d778b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C700-0-0, dag_ret:0, dag_status:2, start_time:1727337660633585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.699801] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=9][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:00.699852] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=5] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.699907] ERROR issue_dba_error (ob_log.cpp:1875) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.699912] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.699923] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=10][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:00.699927] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=4][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:00.699929] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9287d38140, hex: 3230323430393131303133333438, cstr: 20240911013348},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9287d3814e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d380e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3080, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7221, data_checksum:4284205003, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265b00030, buf_size:7221, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1004563202, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:01:00.699946] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=16][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:00.699948] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:00.699950] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:00.699952] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:00.699975] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=7] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265a08060, count:4, allocator:0x7f928643e7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9265a08350, local_datums:0x7f9265a08350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:00.700007] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=28] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264bae060, count:4, allocator:0x7f928643f558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264bae350, local_datums:0x7f9264bae350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:00.699954] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a64350, local_datums:0x7f9264a64350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a643d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a64350, local_datums:0x7f9264a64350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a643d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92c1d9e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c1d9e118, limit_param:null, stmt_allocator:0x7f92c1d9e7d8, allocator:0x7f92c1d9e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a64350, local_datums:0x7f9264a64350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a643d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:01:00.700016] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=62][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:00.700018] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a64350, local_datums:0x7f9264a64350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a643d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92c1d9e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c1d9e118, limit_param:null, stmt_allocator:0x7f92c1d9e7d8, allocator:0x7f92c1d9e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a64350, local_datums:0x7f9264a64350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a643d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a64408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:01:00.700050] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=31][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:00.700052] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=1] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.700066] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=10] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337660633383, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.700076] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:00.700090] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=3][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:00.700094] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:00.700096] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337660699444, end_time=1727337660700096, runtime=652, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FF-0-0, dag_ret:0, dag_status:2, start_time:1727337660633577, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.700105] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FF-0-0, dag_ret:0, dag_status:2, start_time:1727337660633577, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:00.700114] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=66536, dag_cnt=45, dag_cnts_[dag.get_type()]=44, &dag=0x7f9264d76fc0, dag={ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FF-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660633577, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.700126] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FF-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660633573, task_id:YB42C0A8002D-0006230056D8C6FF-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.700284] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:00.700338] ERROR issue_dba_error (ob_log.cpp:1875) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.700342] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.700349] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:00.700352] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:00.700354] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9264bba140, hex: 3230323430393131303133333131, cstr: 20240911013311},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9264bba14e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264bba0e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264bba118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-15038, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7227, data_checksum:1452123993, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265b00030, buf_size:7227, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1818264584, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:01:00.700367] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=13][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:00.700369] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:00.700371] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:00.700373] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:00.700374] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9265a08350, local_datums:0x7f9265a08350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a083d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a08408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9265a08350, local_datums:0x7f9265a08350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a083d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a08408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f928643e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f928643e118, limit_param:null, stmt_allocator:0x7f928643e7d8, allocator:0x7f928643e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:2096135703507, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9265a08350, local_datums:0x7f9265a08350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a083d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a08408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:01:00.700437] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=63][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:00.700439] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9265a08350, local_datums:0x7f9265a08350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a083d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a08408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f928643e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f928643e118, limit_param:null, stmt_allocator:0x7f928643e7d8, allocator:0x7f928643e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:2096135703507, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9265a08350, local_datums:0x7f9265a08350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a083d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a08408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:01:00.700506] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=67][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:00.700510] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=3] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.700516] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] succeed to add sys task(task={start_time:1727337660700513, task_id:YB42C0A8002D-0006230056D8C702-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.700521] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=5] schedule one task(task=0x7f9264d78310, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.700522] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=9] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337660633398, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.700543] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=18][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:00.700557] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=3][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:00.700561] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:00.700563] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337660699927, end_time=1727337660700563, runtime=636, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C700-0-0, dag_ret:0, dag_status:2, start_time:1727337660633585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.700566] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=7][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.700571] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.700574] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.700571] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C700-0-0, dag_ret:0, dag_status:2, start_time:1727337660633585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:00.700582] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.700607] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337660633488, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.700580] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=66994, dag_cnt=44, dag_cnts_[dag.get_type()]=43, &dag=0x7f9264d774f0, dag={ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C700-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660633585, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.700621] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C700-0-0] [lt=38] succeed to del sys task(removed_task={start_time:1727337660633581, task_id:YB42C0A8002D-0006230056D8C700-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.700617] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752755184, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828752755184, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.700630] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.700662] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.700677] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=11] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:226}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752755184, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.700697] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:226}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752755184, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:226}, table_count:1, [{i:0, table_key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}, merge_scn:{val:1726241777443612000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660633668, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=67.01ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.700723] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C701-0-0, dag_ret:0, dag_status:2, start_time:1727337660633591, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1480, output_block_cnt_arr:0x7f92728f1488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:56623104, latest_update_ts:1727337660633668, estimated_finish_time:1727337677700750}) [2024-09-26 16:01:00.700748] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=25] succeed to init merge ctx(task={this:0x7f9264d77de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C701-0-0, dag_ret:0, dag_status:2, start_time:1727337660633591, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:01:00.700755] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=7] task finish process(ret=0, start_time=1727337660633667, end_time=1727337660700755, runtime=67088, *this={this:0x7f9264d77de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C701-0-0, dag_ret:0, dag_status:2, start_time:1727337660633591, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:01:00.700774] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=4] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1, storage_schema:{this:0x7f9263a4b2b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.700782] INFO [STORAGE] prepare_tx_data_list (ob_tx_data_memtable.cpp:266) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=7] construct list more than once. skip this time(this=0x7f92a2294080) [2024-09-26 16:01:00.700926] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.700941] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=14] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337660633362, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=17, max_schema_version_on_memtable=1725120002010696) [2024-09-26 16:01:00.700949] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=17, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.701050] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1887, total_rows=92463) [2024-09-26 16:01:00.701054] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.701060] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200489}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:19, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.701080] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:200489}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200489}, table_count:2, [{i:0, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660633638, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=67.42ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.701378] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FE-0-0, dag_ret:0, dag_status:2, start_time:1727337660633569, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab460, output_block_cnt_arr:0x7f92684ab468, concurrent_cnt:1, estimate_row_cnt:254187, estimate_occupy_size:138412032, latest_update_ts:1727337660633638, estimated_finish_time:1727337677701443}) [2024-09-26 16:01:00.701392] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=14] succeed to init merge ctx(task={this:0x7f9264d76e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FE-0-0, dag_ret:0, dag_status:2, start_time:1727337660633569, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.701398] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=6] task finish process(ret=0, start_time=1727337660633636, end_time=1727337660701398, runtime=67762, *this={this:0x7f9264d76e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FE-0-0, dag_ret:0, dag_status:2, start_time:1727337660633569, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.701583] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=4] schedule one task(task=0x7f9268486030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.701592] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=5] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.701600] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337660701597, task_id:YB42C0A8002D-0006230056D8C703-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.701604] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=4] schedule one task(task=0x7f9264d78840, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.701641] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.701646] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.701648] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.701653] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.701675] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=7] multi version scan iterator init succ(param.table_id_=226, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3c060, count:5, allocator:0x7f92c183a7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3c360, local_datums:0x7f9263a3c360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:01:00.701692] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fc060, count:3, allocator:0x7f928643e7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92665fc320, local_datums:0x7f92665fc320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:01:00.701713] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=17] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c76060, count:3, allocator:0x7f928643f558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9262c76320, local_datums:0x7f9262c76320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:01:00.701879] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:00.701924] ERROR issue_dba_error (ob_log.cpp:1875) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.701928] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.701934] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=5][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:00.701936] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:00.701938] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e8070, hex: 9F094D0300000000, int: 55380383},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e80a8, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e80e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3044, column_count:19, rowkey_column_count:3, has_column_checksum:0, row_count:83, row_store_type:0, opt:5, var_column_count:0, row_offset:16109, original_length:16381, max_merged_trans_version:1726163995623997000, data_length:16381, data_zlength:7759, data_checksum:3416849541, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92665a6030, buf_size:7759, data_size:16381, row_count:83, column_count:19, max_merged_trans_version:1726163995623997000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3537734793, row_count_delta:83, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16381}) [2024-09-26 16:01:00.701950] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=12][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:00.701952] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:00.701953] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:00.701955] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:00.701956] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9262c76320, local_datums:0x7f9262c76320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c763a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9262c764f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c765d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9262c766e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9262c76320, local_datums:0x7f9262c76320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c763a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9262c764f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c765d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9262c766e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f928643ee98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f928643ee98, limit_param:null, stmt_allocator:0x7f928643f558, allocator:0x7f928643f558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9262c76320, local_datums:0x7f9262c76320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c763a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9262c764f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c765d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9262c766e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337660633362, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}) [2024-09-26 16:01:00.702037] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=80][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:00.702039] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9262c76320, local_datums:0x7f9262c76320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c763a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9262c764f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c765d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9262c766e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f928643ee98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f928643ee98, limit_param:null, stmt_allocator:0x7f928643f558, allocator:0x7f928643f558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9262c76320, local_datums:0x7f9262c76320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c763a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9262c764f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c765d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9262c766e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337660633362, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}]) [2024-09-26 16:01:00.702077] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=38][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:00.702080] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=2] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.702093] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=9] partition merge iter row count(i=1, row_count=83, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, table={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337660633362, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.702102] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:00.702113] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:00.702117] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:00.702119] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337660701624, end_time=1727337660702119, runtime=495, *this={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FE-0-0, dag_ret:0, dag_status:2, start_time:1727337660633569, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.702126] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FE-0-0, dag_ret:0, dag_status:2, start_time:1727337660633569, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:00.702133] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=68564, dag_cnt=43, dag_cnts_[dag.get_type()]=42, &dag=0x7f9264d76a90, dag={ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FE-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660633569, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.702152] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FE-0-0] [lt=16] succeed to del sys task(removed_task={start_time:1727337660633566, task_id:YB42C0A8002D-0006230056D8C6FE-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.702212] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337660702209, task_id:YB42C0A8002D-0006230056D8C704-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.702218] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=6] schedule one task(task=0x7f9264d78d70, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.702492] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=8][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.702497] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.702500] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.702504] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.702509] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8621da0, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:480, tablet_addr:[816017-1849-0-2305843619101044736-1](offset=1994752,size=568,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:200004}, data_tablet_id:{id:200004}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1082501607, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1710828761605424, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1710828761605424}, table_store_addr:{ptr:{this:0x7f92b8622434, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844140939550720-0](offset=1974272,size=1054,type=2,seq=0), type:"MAJOR", tablet_id:{id:200004}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077597769620000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843906863834142-0](offset=1975326,size=836,type=2,seq=0), type:"MINOR", tablet_id:{id:200004}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178584585241000, upper_trans_version:1726178584585241000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337660633761, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697961984-1](offset=1982464,size=332,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834499404-0](offset=1982796,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843252955070940-0](offset=1982940,size=227,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828761613515000, v:0}, create_commit_version:1710828761613515000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a74ea0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:01:00.702552] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=43] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.702563] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.704837] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8640e00, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:506, tablet_addr:[816017-1849-0-2305843647017144320-1](offset=806912,size=594,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, data_tablet_id:{id:200489}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725991206240736000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1725991205062061000, cur_report_version:1725991205062061000, data_checksum:1029137533, row_count:46137450}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1725120000848360, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1725120000848360}, table_store_addr:{ptr:null, addr:[816017-1849-0-2305845308095737856-1](offset=798720,size=2141,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718219357-0](offset=800861,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255101372665-0](offset=801017,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1725991205062061000}, medium_info_list:{ptr:(nil), addr:[816017-1849-0-2305843505283219456-0](offset=802816,size=462,type=2,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1725120001569133000, v:0}, create_commit_version:1725120001569133000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a75670, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:01:00.704902] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=65] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:01:00.704914] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.704934] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337660633761, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.704945] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828761605424, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828761605424, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.704960] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.704963] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.704971] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200004}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1710828761605424, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.705001] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=23] succeed to build merge ctx(tablet_id={id:200004}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828761605424, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200004}, table_count:1, [{i:0, table_key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8621da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660702486, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.48ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.705027] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C704-0-0, dag_ret:0, dag_status:2, start_time:1727337660702218, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685e7460, output_block_cnt_arr:0x7f92685e7468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337660702486, estimated_finish_time:1727337677705030}) [2024-09-26 16:01:00.705034] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d78d70, type:15, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C704-0-0, dag_ret:0, dag_status:2, start_time:1727337660702218, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.705040] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=5] task finish process(ret=0, start_time=1727337660702484, end_time=1727337660705040, runtime=2556, *this={this:0x7f9264d78d70, type:15, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C704-0-0, dag_ret:0, dag_status:2, start_time:1727337660702218, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.705091] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=3] schedule one task(task=0x7f926585e3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.705160] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=200004, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263be8060, count:3, allocator:0x7f928643e7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263be8320, local_datums:0x7f9263be8320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:00.705322] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:00.705364] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.705368] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.705375] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:00.705377] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:00.705379] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264dd6070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264dd60a8, hex: C852A9F488570BE8, int: -1726189786434547000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264dd60e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19071, column_count:8, rowkey_column_count:3, has_column_checksum:0, row_count:218, row_store_type:0, opt:0, var_column_count:0, row_offset:15549, original_length:16361, max_merged_trans_version:1726190661653507000, data_length:16361, data_zlength:5996, data_checksum:2604580500, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9265892030, buf_size:5996, data_size:16361, row_count:218, column_count:8, max_merged_trans_version:1726190661653507000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:83488362, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16361}) [2024-09-26 16:01:00.705400] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=20][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:00.705402] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:00.705403] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:00.705405] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:00.705406] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263be8320, local_datums:0x7f9263be8320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be83a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263be8320, local_datums:0x7f9263be8320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be83a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f928643e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f928643e118, limit_param:null, stmt_allocator:0x7f928643e7d8, allocator:0x7f928643e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3904677171908654128, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263be8320, local_datums:0x7f9263be8320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be83a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337660633761, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:01:00.705492] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=86][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:00.705494] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263be8320, local_datums:0x7f9263be8320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be83a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f928643e118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f928643e118, limit_param:null, stmt_allocator:0x7f928643e7d8, allocator:0x7f928643e7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3904677171908654128, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263be8320, local_datums:0x7f9263be8320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be83a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be8480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337660633761, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:01:00.705530] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=36][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:00.705533] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=2] partition merge iter row count(i=0, row_count=218, ghost_row_count=0, pkey={tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337660633761, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.705545] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:00.705556] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:00.705559] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:00.705561] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337660705104, end_time=1727337660705561, runtime=457, *this={this:0x7f926585e3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C704-0-0, dag_ret:0, dag_status:2, start_time:1727337660702218, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.705567] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926585e3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C704-0-0, dag_ret:0, dag_status:2, start_time:1727337660702218, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:00.705575] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=3357, dag_cnt=42, dag_cnts_[dag.get_type()]=41, &dag=0x7f9264d789b0, dag={ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C704-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660702218, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.705583] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C704-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660702209, task_id:YB42C0A8002D-0006230056D8C704-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.705610] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337660705608, task_id:YB42C0A8002D-0006230056D8C705-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.705614] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=3] schedule one task(task=0x7f9264d792a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.705648] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.705655] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.705665] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=9][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.705672] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.705676] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861eec0, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:493, tablet_addr:[816017-1849-0-2305843633059586048-2](offset=1892352,size=581,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1}, data_tablet_id:{id:1}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:778798285, row_count:7249}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861f59c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844357835309056-0](offset=1884160,size=1256,type=2,seq=0), type:"MAJOR", tablet_id:{id:1}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077591663360001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843964845802728-0](offset=1885416,size=890,type=2,seq=0), type:"MINOR", tablet_id:{id:1}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178578410947000, upper_trans_version:1726178578410947000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337660633782, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcd4f0, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241621541056000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405180}, this:0x7f92a3bcd4f0, timestamp:1727334209405180, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:53, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277807268534000, v:0}, rec_scn:{val:1726241631569302000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771609600-2](offset=1888256,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843169203114317-0](offset=1888589,size=149,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881234914-0](offset=1888738,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828751995331000, v:0}, create_commit_version:1710828751995331000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a76610, 0x7f92a3bcd4f0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.705725] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=49] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.705735] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241621541056000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.706212] INFO [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:329) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=3] pre-process commit versions row(get_ls_id()={id:1}, do_recycle_=true, current_time=1727337660706210, prev_recycle_time=0) [2024-09-26 16:01:00.708512] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b863bfe0, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:506, tablet_addr:[816017-1849-0-2305843647017062400-1](offset=724992,size=594,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, data_tablet_id:{id:200489}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725991206240736000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1725991205062061000, cur_report_version:1725991205062061000, data_checksum:2143506547, row_count:46137450}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1725120000848352, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1725120000848352}, table_store_addr:{ptr:null, addr:[816017-1849-0-2305845196426506240-1](offset=716800,size=2037,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718137333-0](offset=718837,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843255101290641-0](offset=718993,size=229,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1725991205062061000}, medium_info_list:{ptr:(nil), addr:[816017-1849-0-2305843520315523072-0](offset=720896,size=476,type=2,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1725120001569133000, v:0}, create_commit_version:1725120001569133000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a746d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:01:00.708546] INFO [STORAGE] get_ls_min_end_scn (ob_ls_tablet_service.cpp:5618) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=19] get ls min end scn finish(ls_checkpoint={val:1726163586617842000, v:0}) [2024-09-26 16:01:00.708550] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=37] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:01:00.708567] WDIAG [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:761) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=6][errcode=-4023] failed to get snapshot version(ret=-4023, MTL_ID()=1) [2024-09-26 16:01:00.708572] INFO [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:773) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=5] get tx data recycle scn finish.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, recycle_scn={val:1726076389100953000, v:0}, min_end_scn_from_old_tablets={val:4611686018427387903, v:0}, min_end_scn_from_latest_tablets={val:1726076389100953000, v:0}) [2024-09-26 16:01:00.708578] WDIAG [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:336) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=5][errcode=-4023] get recycle ts failed.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:01:00.708579] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.708583] WDIAG [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:236) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=1][errcode=-4023] process commit version row failed.(ret=-4023, ret="OB_EAGAIN", this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337660633251, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:01:00.708595] INFO [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:254) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=11] [TX DATA MERGE]pre process for merge done(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}, this=0x7f92a2294080, tg=time guard have no click events for optimization, fake_tx_data_guard={tx_data:{tx_id:{txid:9223372036854775807}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337660633251, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:01:00.708607] WDIAG [STORAGE.COMPACTION] pre_process_tx_data_table_merge_ (ob_tx_table_merge_task.cpp:117) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=11][errcode=-4023] do pre process for tx data table merge failed.(ret=-4023, ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, table={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337660633251, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:01:00.708618] WDIAG [STORAGE.COMPACTION] inner_init_ctx (ob_tx_table_merge_task.cpp:157) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=10][errcode=-4023] pre process tx data table for merge failed.(ret=-4023, ret="OB_EAGAIN", ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.708620] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27] success to notify_need_writing_throttling True [2024-09-26 16:01:00.708625] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.708622] WDIAG [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1061) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=2][errcode=-4023] fail to inner init ctx(ret=-4023, tablet_id={id:49402}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, create_snapshot_version:0, is_full_merge:false, merge_level:1, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:4, range_array:[], concurrent_cnt:0, is_inited:false}, schema_ctx:{schema_version:1, storage_schema:{this:0x7f9263a4b2b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:49402}, table_count:1, [{i:0, table_key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref:4}]}, schedule_major:false, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}, merge_scn:{val:4611686018427387903, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:NULL, compaction_filter:NULL, time_guard:total=75.05ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}) [2024-09-26 16:01:00.708646] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:975) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=24][errcode=-4023] failed to build merge ctx(ret=-4023, ctx->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.708650] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:994) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=3][errcode=-4023] sstable merge finish(ret=-4023, ctx=0x7f9263a4a060, task={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C6FC-0-0, dag_ret:0, dag_status:2, start_time:1727337660633533, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:01:00.708657] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=7][errcode=-4023] failed to process task(ret=-4023) [2024-09-26 16:01:00.708660] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=3] task finish process(ret=-4023, start_time=1727337660633562, end_time=1727337660708660, runtime=75098, *this={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C6FC-0-0, dag_ret:0, dag_status:2, start_time:1727337660633533, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:01:00.708667] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=6][errcode=-4023] failed to do work(ret=-4023, *task_={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C6FC-0-0, dag_ret:0, dag_status:2, start_time:1727337660633533, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:00.708686] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=17] dag finished(dag_ret=-4023, runtime=75153, dag_cnt=41, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264d76030, dag={ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C6FC-0-0, dag_ret:-4023, dag_status:5, start_time:1727337660633533, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}) [2024-09-26 16:01:00.708698] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2864][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C6FC-0-0] [lt=8] succeed to del sys task(removed_task={start_time:1727337660633523, task_id:YB42C0A8002D-0006230056D8C6FC-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.708726] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726158220122116000) [2024-09-26 16:01:00.708735] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:00.708739] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=7] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.708771] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337660708768, task_id:YB42C0A8002D-0006230056D8C706-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.708777] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=5] schedule one task(task=0x7f9264d797d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.708784] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.708788] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.708796] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:00.708798] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:00.708802] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.708806] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.708808] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.708800] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a0a8, hex: 0000000000000000, int: 0},idx=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3a0e0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a118, hex: 58A3C56A4A280BE8, int: -1726241732082425000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3a150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-19590, column_count:10, rowkey_column_count:5, has_column_checksum:0, row_count:224, row_store_type:0, opt:0, var_column_count:0, row_offset:15527, original_length:16363, max_merged_trans_version:1726241777443612000, data_length:16363, data_zlength:16363, data_checksum:2057046089, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6804090, buf_size:16363, data_size:16363, row_count:224, column_count:10, max_merged_trans_version:1726241777443612000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1006468742, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16363}) [2024-09-26 16:01:00.708813] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.708813] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=13][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:00.708816] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:00.708817] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:00.708816] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.708818] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:01:00.708820] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:00.708821] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.708822] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:00.708823] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3c360, local_datums:0x7f9263a3c360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3c3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3c360, local_datums:0x7f9263a3c360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3c3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f92c183a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c183a118, limit_param:null, stmt_allocator:0x7f92c183a7d8, allocator:0x7f92c183a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3c360, local_datums:0x7f9263a3c360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3c3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337660633488, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}) [2024-09-26 16:01:00.708890] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=66][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:00.708892] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3c360, local_datums:0x7f9263a3c360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3c3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f92c183a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92c183a118, limit_param:null, stmt_allocator:0x7f92c183a7d8, allocator:0x7f92c183a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3c360, local_datums:0x7f9263a3c360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f9263a3c3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a3c450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337660633488, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}]) [2024-09-26 16:01:00.708926] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=34][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:00.708928] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=1] partition merge iter row count(i=0, row_count=224, ghost_row_count=0, pkey={tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, table={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337660633488, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.708942] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:00.708956] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:00.708960] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:00.708961] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337660701614, end_time=1727337660708961, runtime=7347, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C701-0-0, dag_ret:0, dag_status:2, start_time:1727337660633591, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:01:00.708968] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C701-0-0, dag_ret:0, dag_status:2, start_time:1727337660633591, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:00.708975] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=75384, dag_cnt=40, dag_cnts_[dag.get_type()]=40, &dag=0x7f9264d77a20, dag={ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C701-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660633591, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}) [2024-09-26 16:01:00.708993] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C701-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660633588, task_id:YB42C0A8002D-0006230056D8C701-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.709053] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337660633525, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.709063] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=9] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:01:00.709066] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9263b5d480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.709183] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3223, total_rows=22561) [2024-09-26 16:01:00.709187] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.709194] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263b5d480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847409}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.709217] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:1152921504606847409}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9263b5d480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847409}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8640e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660701632, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=7.56ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.709235] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337660709232, task_id:YB42C0A8002D-0006230056D8C707-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.709243] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=7] schedule one task(task=0x7f9264d79d00, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.709266] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.709269] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.709273] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.709278] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.709281] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8632d40, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[816017-1849-0-2305843650239590400-2](offset=2027520,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:444}, data_tablet_id:{id:444}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1509715702, row_count:4}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8633440, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844406153818112-0](offset=2011136,size=1301,type=2,seq=0), type:"MAJOR", tablet_id:{id:444}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077198228842001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843949813544213-0](offset=2012437,size=876,type=2,seq=0), type:"MINOR", tablet_id:{id:444}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726177998310996000, upper_trans_version:1726177998310996000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337660633814, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bce490, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209461185}, this:0x7f92a3bce490, timestamp:1727334209461185, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:54, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277298911712000, v:0}, rec_scn:{val:1726241718316746000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697998848-2](offset=2019328,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161889100-0](offset=2019660,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881365998-0](offset=2019822,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447886032-0](offset=2020048,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a775b0, 0x7f92a3bce490, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.709344] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=62] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.709351] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.709351] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C703-0-0, dag_ret:0, dag_status:2, start_time:1727337660701604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263b5d5d0, output_block_cnt_arr:0x7f9263b5d5d8, concurrent_cnt:1, estimate_row_cnt:32439, estimate_occupy_size:18874368, latest_update_ts:1727337660701632, estimated_finish_time:1727337677709359}) [2024-09-26 16:01:00.709360] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264d78840, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C703-0-0, dag_ret:0, dag_status:2, start_time:1727337660701604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.709365] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=5] task finish process(ret=0, start_time=1727337660701629, end_time=1727337660709365, runtime=7736, *this={this:0x7f9264d78840, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C703-0-0, dag_ret:0, dag_status:2, start_time:1727337660701604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.709397] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=3] schedule one task(task=0x7f9272926030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.709460] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=1152921504606847409, range=0x7f92aa952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:5, allocator:0x7f92ba7fc7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:01:00.709700] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=20][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:00.709748] ERROR issue_dba_error (ob_log.cpp:1875) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.709754] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.709762] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:00.709764] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:00.709766] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 7, flag: 0, null: 0, ptr: 0x7f92665fe178, hex: 31303033343330, cstr: 1003430},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f92665fe17f, hex: 3230323430393133303435303537, cstr: 20240913045057},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe0e0, hex: 6E48500300000000, int: 55593070},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe118, hex: 58B03BC5A1610BE8, int: -1726178684739801000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-26357, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:250, row_store_type:0, opt:5, var_column_count:0, row_offset:15394, original_length:16334, max_merged_trans_version:1726190643638412000, data_length:16334, data_zlength:5786, data_checksum:2846467531, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92684de030, buf_size:5786, data_size:16334, row_count:250, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:796410710, row_count_delta:250, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16334}) [2024-09-26 16:01:00.709780] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=14][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:00.709782] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:00.709783] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:00.709785] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:00.709786] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92ba7fc118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92ba7fc118, limit_param:null, stmt_allocator:0x7f92ba7fc7d8, allocator:0x7f92ba7fc7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337660633525, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:01:00.709840] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=54][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:00.709841] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92ba7fc118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92ba7fc118, limit_param:null, stmt_allocator:0x7f92ba7fc7d8, allocator:0x7f92ba7fc7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337660633525, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:01:00.709874] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=33][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:00.709876] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=1] partition merge iter row count(i=0, row_count=250, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337660633525, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.709888] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:00.709898] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:00.709902] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:00.709903] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337660709410, end_time=1727337660709903, runtime=493, *this={this:0x7f9272926030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C703-0-0, dag_ret:0, dag_status:2, start_time:1727337660701604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.709912] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=9][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9272926030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C703-0-0, dag_ret:0, dag_status:2, start_time:1727337660701604, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:00.709920] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=8316, dag_cnt=39, dag_cnts_[dag.get_type()]=39, &dag=0x7f9264d78480, dag={ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C703-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660701604, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.709928] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C703-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660701597, task_id:YB42C0A8002D-0006230056D8C703-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.709952] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337660709951, task_id:YB42C0A8002D-0006230056D8C708-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.709955] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=3] schedule one task(task=0x7f9264d7a230, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.710036] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.710041] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.710044] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.710050] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.710053] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.710060] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241316899724000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.710438] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337660633506, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.710492] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=32] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:01:00.710497] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=4] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92685eb480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.710637] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=30] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3214, total_rows=22498) [2024-09-26 16:01:00.710643] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.710650] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92685eb480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847408}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.710671] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:1152921504606847408}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92685eb480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847408}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660700553, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=10.09ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.710784] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C702-0-0, dag_ret:0, dag_status:2, start_time:1727337660700521, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685eb5d0, output_block_cnt_arr:0x7f92685eb5d8, concurrent_cnt:1, estimate_row_cnt:35098, estimate_occupy_size:18874368, latest_update_ts:1727337660700553, estimated_finish_time:1727337677710793}) [2024-09-26 16:01:00.710805] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=20] succeed to init merge ctx(task={this:0x7f9264d78310, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C702-0-0, dag_ret:0, dag_status:2, start_time:1727337660700521, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.710811] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=6] task finish process(ret=0, start_time=1727337660700551, end_time=1727337660710811, runtime=10260, *this={this:0x7f9264d78310, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C702-0-0, dag_ret:0, dag_status:2, start_time:1727337660700521, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.710845] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337660633782, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.710857] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:0, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.710854] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337660633798, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.710868] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.710871] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.710866] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=12] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753247280, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828753247280, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.710881] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.710885] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.710875] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:0, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.710893] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:412}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753247280, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.710893] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:1}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:0, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1}, table_count:1, [{i:0, table_key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}, merge_scn:{val:1726241621541056000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660705640, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.23ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.710913] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C705-0-0, dag_ret:0, dag_status:2, start_time:1727337660705613, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a7480, output_block_cnt_arr:0x7f92a23a7488, concurrent_cnt:1, estimate_row_cnt:3, estimate_occupy_size:6291456, latest_update_ts:1727337660705640, estimated_finish_time:1727337677710916}) [2024-09-26 16:01:00.710919] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d792a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C705-0-0, dag_ret:0, dag_status:2, start_time:1727337660705613, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:01:00.710923] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=4] task finish process(ret=0, start_time=1727337660705638, end_time=1727337660710923, runtime=5285, *this={this:0x7f9264d792a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C705-0-0, dag_ret:0, dag_status:2, start_time:1727337660705613, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:01:00.710912] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:412}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753247280, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:412}, table_count:1, [{i:0, table_key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660708795, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.10ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.710938] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C706-0-0, dag_ret:0, dag_status:2, start_time:1727337660708776, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b470, output_block_cnt_arr:0x7f9263a4b478, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337660708795, estimated_finish_time:1727337677710940}) [2024-09-26 16:01:00.710945] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d797d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C706-0-0, dag_ret:0, dag_status:2, start_time:1727337660708776, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.710950] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=5] task finish process(ret=0, start_time=1727337660708792, end_time=1727337660710950, runtime=2158, *this={this:0x7f9264d797d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C706-0-0, dag_ret:0, dag_status:2, start_time:1727337660708776, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.710966] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=3] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.710972] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=4] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.710975] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=1] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.711022] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=412, range=0x7f92aaf52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:7}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fc060, count:4, allocator:0x7f92b32387d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f92665fc350, local_datums:0x7f92665fc350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:01:00.711048] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=13][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:00.711090] ERROR issue_dba_error (ob_log.cpp:1875) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.711094] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.711104] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=9][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:00.711107] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:00.711109] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 23, flag: 0, null: 0, ptr: 0x7f9262c76140, hex: 6865617274626561745F736572766963655F65706F6368, cstr: heartbeat_service_epoch},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c760e0, hex: 8808AF9C9C5C0BE8, int: -1726184204453083000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-7627, column_count:7, rowkey_column_count:4, has_column_checksum:0, row_count:1, row_store_type:0, opt:5, var_column_count:0, row_offset:136, original_length:80, max_merged_trans_version:1726184204453083000, data_length:80, data_zlength:80, data_checksum:1549639079, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f926f404090, buf_size:80, data_size:80, row_count:1, column_count:7, max_merged_trans_version:1726184204453083000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:37835160, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:80}) [2024-09-26 16:01:00.711122] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:00.711124] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=1][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:01:00.711126] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=2][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:01:00.711128] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=2][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:01:00.711129] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=1][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:01:00.711132] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337660633798, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.711146] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=11][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:00.711154] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:00.711158] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:00.711160] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337660710985, end_time=1727337660711159, runtime=174, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C706-0-0, dag_ret:0, dag_status:2, start_time:1727337660708776, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.711167] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C706-0-0, dag_ret:0, dag_status:2, start_time:1727337660708776, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:00.711175] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=6] dag finished(dag_ret=-4184, runtime=2399, dag_cnt=38, dag_cnts_[dag.get_type()]=38, &dag=0x7f9264d79410, dag={ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C706-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660708776, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.711183] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C706-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660708768, task_id:YB42C0A8002D-0006230056D8C706-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.711259] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1152921504606847408, range=0x7f92a9d52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:5, allocator:0x7f9269f787d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:01:00.711494] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=18][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:00.711531] ERROR issue_dba_error (ob_log.cpp:1875) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.711534] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.711541] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:00.711543] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:00.711544] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9263b5a178, hex: 30323232362D32312D373825, cstr: 02226-21-78%},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9263b5a184, hex: 3230323430393133303435343039, cstr: 20240913045409},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a0e0, hex: CD7F500300000000, int: 55607245},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a118, hex: E8EC05CCA0600BE8, int: -1726179788432479000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5a150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-11869, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:238, row_store_type:0, opt:5, var_column_count:0, row_offset:15429, original_length:16321, max_merged_trans_version:1726190643638412000, data_length:16321, data_zlength:5388, data_checksum:152456761, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9264d58030, buf_size:5388, data_size:16321, row_count:238, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2114797869, row_count_delta:238, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16321}) [2024-09-26 16:01:00.711570] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=26][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:00.711572] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:00.711574] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:00.711575] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:00.711577] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a383e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a383e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9269f78118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269f78118, limit_param:null, stmt_allocator:0x7f9269f787d8, allocator:0x7f9269f787d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1073741824, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a383e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337660633506, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:01:00.711631] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=53][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:00.711632] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a383e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9269f78118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269f78118, limit_param:null, stmt_allocator:0x7f9269f787d8, allocator:0x7f9269f787d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1073741824, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a383e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263a38450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337660633506, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:01:00.711664] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=32][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:00.711666] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=2] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337660633506, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.711678] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:00.711687] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:00.711690] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:00.711692] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337660711209, end_time=1727337660711692, runtime=483, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C702-0-0, dag_ret:0, dag_status:2, start_time:1727337660700521, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.711699] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C702-0-0, dag_ret:0, dag_status:2, start_time:1727337660700521, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:00.711706] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=11185, dag_cnt=37, dag_cnts_[dag.get_type()]=37, &dag=0x7f9264d77f50, dag={ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C702-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660700521, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.711714] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C702-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660700513, task_id:YB42C0A8002D-0006230056D8C702-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.711784] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=1, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:8}, cols_desc:{ObIArray:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685ea060, count:5, allocator:0x7f92636147d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92685ea360, local_datums:0x7f92685ea360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:00.712170] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:00.712227] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=8][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.712231] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.712237] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:00.712239] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:00.712241] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92665fe178, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe0a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92665fe189, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe118, hex: 60D2640CCC290BE8, int: -1726240075808452000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-20032, column_count:8, rowkey_column_count:5, has_column_checksum:0, row_count:158, row_store_type:0, opt:0, var_column_count:0, row_offset:15849, original_length:16421, max_merged_trans_version:1726241621541056000, data_length:16421, data_zlength:16421, data_checksum:3599888316, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6604090, buf_size:16421, data_size:16421, row_count:158, column_count:8, max_merged_trans_version:1726241621541056000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1714289658, row_count_delta:2, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16421}) [2024-09-26 16:01:00.712251] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=10][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:00.712253] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:00.712254] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:00.712256] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=2][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:01:00.712260] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=4][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:00.712261] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:00.712280] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660712277, task_id:YB42C0A8002D-0006230056D8C709-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.712286] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=5] schedule one task(task=0x7f9264d7a760, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.712294] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337660712293, task_id:YB42C0A8002D-0006230056D8C70A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.712297] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=2] schedule one task(task=0x7f9264d7ac90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.712263] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92685ea360, local_datums:0x7f92685ea360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92685ea360, local_datums:0x7f92685ea360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f9263614118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263614118, limit_param:null, stmt_allocator:0x7f92636147d8, allocator:0x7f92636147d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92685ea360, local_datums:0x7f92685ea360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337660633782, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:01:00.712341] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=77][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:00.712343] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92685ea360, local_datums:0x7f92685ea360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f9263614118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263614118, limit_param:null, stmt_allocator:0x7f92636147d8, allocator:0x7f92636147d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92685ea360, local_datums:0x7f92685ea360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea3a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337660633782, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:01:00.712378] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=35][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:00.712380] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=1] partition merge iter row count(i=0, row_count=158, ghost_row_count=0, pkey={tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, table={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337660633782, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.712398] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.712401] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=18][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:00.712401] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.712403] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.712408] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:00.712410] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=5] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.712412] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:00.712413] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337660711756, end_time=1727337660712413, runtime=657, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C705-0-0, dag_ret:0, dag_status:2, start_time:1727337660705613, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:01:00.712420] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C705-0-0, dag_ret:0, dag_status:2, start_time:1727337660705613, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:00.712426] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=6813, dag_cnt=36, dag_cnts_[dag.get_type()]=36, &dag=0x7f9264d78ee0, dag={ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C705-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660705613, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}) [2024-09-26 16:01:00.712433] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C705-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660705608, task_id:YB42C0A8002D-0006230056D8C705-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.712414] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8627b60, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:523, tablet_addr:[816017-1849-0-2305843665271992320-2](offset=2043904,size=611,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:458}, data_tablet_id:{id:458}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2897668175, row_count:77572}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86282a8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844621975949312-0](offset=2035712,size=1502,type=2,seq=0), type:"MAJOR", tablet_id:{id:458}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389055265000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844927992370654-0](offset=2037214,size=1787,type=2,seq=0), type:"MINOR", tablet_id:{id:458}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178190631204000, upper_trans_version:1726178190631204000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337660633863, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd1b40, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209496526}, this:0x7f92a3bd1b40, timestamp:1727334209496526, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:61, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274225933800000, v:0}, rec_scn:{val:1726241806895572002, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771761152-2](offset=2039808,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046811469-0](offset=2040141,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881386491-0](offset=2040315,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447906525-0](offset=2040541,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7a490, 0x7f92a3bd1b40, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.712470] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=56] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.712477] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.712498] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.712501] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.712503] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.712506] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.712508] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.712512] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241465393532000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.712521] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337660712519, task_id:YB42C0A8002D-0006230056D8C70B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.712524] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=3] schedule one task(task=0x7f9264d7b1c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.712552] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.712556] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.712558] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.712562] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.712566] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b863cf80, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:522, tablet_addr:[816017-1849-0-2305843664198197248-2](offset=1990656,size=610,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:487}, data_tablet_id:{id:487}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1549633270, row_count:11693}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863d6c8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844647745695744-0](offset=1978368,size=1526,type=2,seq=0), type:"MAJOR", tablet_id:{id:487}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389060348000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843977730799094-0](offset=1979894,size=902,type=2,seq=0), type:"MINOR", tablet_id:{id:487}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178190844721000, upper_trans_version:1726178190844721000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337660633882, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd4850, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209506557}, this:0x7f92a3bd4850, timestamp:1727334209506557, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:66, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274231198751000, v:0}, rec_scn:{val:1726241813664082000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843367845449728-2](offset=1986560,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046758222-0](offset=1986894,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881333244-0](offset=1987068,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447853278-0](offset=1987294,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a79cc0, 0x7f92a3bd4850, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.712620] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=53] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.712627] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.713271] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.713319] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=47] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337660633318, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=18, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:01:00.713326] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=6] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.713326] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337660633831, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.713352] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=25] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752918752, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828752918752, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.713380] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=12] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.713385] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.713392] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:309}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752918752, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.713413] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:309}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752918752, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:309}, table_count:1, [{i:0, table_key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}, merge_scn:{val:1726241316899724000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8631da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660710029, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.36ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.713442] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=22] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C708-0-0, dag_ret:0, dag_status:2, start_time:1727337660709955, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad490, output_block_cnt_arr:0x7f92684ad498, concurrent_cnt:1, estimate_row_cnt:6, estimate_occupy_size:6291456, latest_update_ts:1727337660710029, estimated_finish_time:1727337677713445}) [2024-09-26 16:01:00.713467] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=24] succeed to init merge ctx(task={this:0x7f9264d7a230, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C708-0-0, dag_ret:0, dag_status:2, start_time:1727337660709955, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:01:00.713473] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=5] task finish process(ret=0, start_time=1727337660710027, end_time=1727337660713472, runtime=3445, *this={this:0x7f9264d7a230, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C708-0-0, dag_ret:0, dag_status:2, start_time:1727337660709955, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:01:00.713490] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=2] schedule one task(task=0x7f9268486030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.713532] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=309, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:11}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fc060, count:7, allocator:0x7f92b321a7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92665fc3b0, local_datums:0x7f92665fc3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:01:00.713559] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:01:00.713564] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.713571] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200240}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.713589] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:200240}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200240}, table_count:2, [{i:0, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8637b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660633610, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=79.96ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.713691] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:00.713747] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=19][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.713751] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.713759] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:00.713761] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:00.713763] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:7, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 12, flag: 0, null: 0, ptr: 0x7f9262c761e8, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c760e0, hex: 420B000000000000, int: 2882},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76118, hex: 0100000000000000, int: 1, num_digit0: 0},idx=4:{len: 16, flag: 0, null: 0, ptr: 0x7f9262c761f4, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c76188, hex: C818154EB7520BE8, int: -1726195084923955000},idx=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c761c0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19292, column_count:11, rowkey_column_count:7, has_column_checksum:0, row_count:156, row_store_type:0, opt:0, var_column_count:0, row_offset:15827, original_length:16391, max_merged_trans_version:1726241316783508000, data_length:16391, data_zlength:16391, data_checksum:3629777942, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b7204090, buf_size:16391, data_size:16391, row_count:156, column_count:11, max_merged_trans_version:1726241316783508000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:619385244, row_count_delta:1, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16391}) [2024-09-26 16:01:00.713776] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=13][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:00.713778] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:00.713779] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:00.713781] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=2][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:01:00.713783] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:00.713785] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:00.713802] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=18] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:01:00.713786] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92665fc3b0, local_datums:0x7f92665fc3b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc4d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5f0, hex: 9C4DB16500000000, int: 1706118556},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:309}, iter_end:false, schema_rowkey_column_cnt:5, schema_version:1710828752918752, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92665fc3b0, local_datums:0x7f92665fc3b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc4d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5f0, hex: 9C4DB16500000000, int: 1706118556},]}, store_ctx:{this:0x7f92b321a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b321a118, limit_param:null, stmt_allocator:0x7f92b321a7d8, allocator:0x7f92b321a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241316899724000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:140266759512216, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92665fc3b0, local_datums:0x7f92665fc3b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc4d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5f0, hex: 9C4DB16500000000, int: 1706118556},]}, key:{"BIGINT":1},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f9279a04108 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x7f927a839198 latest_compact_node=0x7f927aa30240 max_trans_version={val:1726241316783508000, v:0} max_trans_id=125581128 max_elr_trans_version={val:1726241316783508000, v:0} max_elr_trans_id=125581128 latest_compact_ts=1727334209251890 last_compact_cnt=189 total_trans_node_cnt=418 max_modify_scn={val:1726241316783508000, v:0} min_modify_scn={val:1726178872697529000, v:0}}, version_iter:this=0x7f927a8390e8 trans_version={val:1726241316783508000, v:0} scn={val:1726241316783508000, v:0} tx_id={txid:125581128} prev=0x7f927a838f78 next=0x7f927a839198 modify_count=196 acc_checksum=432508559 version=1726211894725631 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:65} seq_no=1726241316536989, multi_version_iter:this=0x7f927aa0b360 trans_version={val:1726194784763860000, v:0} scn={val:1726194784763860000, v:0} tx_id={txid:125216537} prev=0x7f927aa0b2b0 next=0x7f927aa0b9f8 modify_count=107 acc_checksum=2694663391 version=1726178872309143 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no=1726194784741342, max_committed_trans_version:1726241316783508000, cur_trans_version:{val:1726195084923955000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241316899724000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:156, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337660633831, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:11, cur_pos:0, count:0}}) [2024-09-26 16:01:00.713867] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=80][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:00.713869] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:309}, iter_end:false, schema_rowkey_column_cnt:5, schema_version:1710828752918752, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92665fc3b0, local_datums:0x7f92665fc3b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc4d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5f0, hex: 9C4DB16500000000, int: 1706118556},]}, store_ctx:{this:0x7f92b321a118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b321a118, limit_param:null, stmt_allocator:0x7f92b321a7d8, allocator:0x7f92b321a7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241316899724000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:140266759512216, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726241316783508000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92665fc3b0, local_datums:0x7f92665fc3b0}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3c0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 12, flag: 0, null: 0, ptr: 0x7f927aa0bb1d, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc430, hex: 420B000000000000, int: 2882},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc468, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=4:{len: 16, flag: 0, null: 0, ptr: 0x7f927aa0bb2c, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc4d8, hex: C818154EB7520BE8, int: -1726195084923955000},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc510, hex: 0000000000000000, int: 0},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb3c, hex: BD00336CFD130600, int: 1710829023199421},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f927aa0bb44, hex: F821D91CF7210600, int: 1726195084894712},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5b8, hex: 9C4DB16500000000, int: 1706118556},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc5f0, hex: 9C4DB16500000000, int: 1706118556},]}, key:{"BIGINT":1},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT":2882},{"BIGINT":1},{"VARCHAR":"tenant clog data", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f9279a04108 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=4 list_head=0x7f927a839198 latest_compact_node=0x7f927aa30240 max_trans_version={val:1726241316783508000, v:0} max_trans_id=125581128 max_elr_trans_version={val:1726241316783508000, v:0} max_elr_trans_id=125581128 latest_compact_ts=1727334209251890 last_compact_cnt=189 total_trans_node_cnt=418 max_modify_scn={val:1726241316783508000, v:0} min_modify_scn={val:1726178872697529000, v:0}}, version_iter:this=0x7f927a8390e8 trans_version={val:1726241316783508000, v:0} scn={val:1726241316783508000, v:0} tx_id={txid:125581128} prev=0x7f927a838f78 next=0x7f927a839198 modify_count=196 acc_checksum=432508559 version=1726211894725631 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:65} seq_no=1726241316536989, multi_version_iter:this=0x7f927aa0b360 trans_version={val:1726194784763860000, v:0} scn={val:1726194784763860000, v:0} tx_id={txid:125216537} prev=0x7f927aa0b2b0 next=0x7f927aa0b9f8 modify_count=107 acc_checksum=2694663391 version=1726178872309143 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:89} seq_no=1726194784741342, max_committed_trans_version:1726241316783508000, cur_trans_version:{val:1726195084923955000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241316899724000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:156, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337660633831, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:11, cur_pos:0, count:0}}]) [2024-09-26 16:01:00.713919] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=50][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:00.713922] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=2] partition merge iter row count(i=0, row_count=156, ghost_row_count=0, pkey={tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, table={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337660633831, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.713935] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:00.713944] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:00.713947] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:00.713949] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337660713505, end_time=1727337660713949, runtime=444, *this={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C708-0-0, dag_ret:0, dag_status:2, start_time:1727337660709955, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:01:00.713956] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C708-0-0, dag_ret:0, dag_status:2, start_time:1727337660709955, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:00.713963] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=4008, dag_cnt=35, dag_cnts_[dag.get_type()]=35, &dag=0x7f9264d79e70, dag={ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C708-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660709955, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}) [2024-09-26 16:01:00.713971] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C708-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660709951, task_id:YB42C0A8002D-0006230056D8C708-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.714017] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337660714016, task_id:YB42C0A8002D-0006230056D8C70C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.714023] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=6] schedule one task(task=0x7f9264d7b6f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.714310] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FD-0-0, dag_ret:0, dag_status:2, start_time:1727337660633556, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a43460, output_block_cnt_arr:0x7f9263a43468, concurrent_cnt:1, estimate_row_cnt:85417, estimate_occupy_size:134217728, latest_update_ts:1727337660633610, estimated_finish_time:1727337677714374}) [2024-09-26 16:01:00.714321] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=11] succeed to init merge ctx(task={this:0x7f9264d76920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FD-0-0, dag_ret:0, dag_status:2, start_time:1727337660633556, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.714327] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=6] task finish process(ret=0, start_time=1727337660633607, end_time=1727337660714327, runtime=80720, *this={this:0x7f9264d76920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FD-0-0, dag_ret:0, dag_status:2, start_time:1727337660633556, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.714354] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=4] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.714358] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.714361] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.714372] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=11][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.714377] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.714380] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861cf80, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:528, tablet_addr:[816017-1849-0-2305843670640488448-2](offset=1830912,size=616,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:455}, data_tablet_id:{id:455}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4242650745, row_count:15909}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861d6ec, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305845137371811840-0](offset=1822720,size=1982,type=2,seq=0), type:"MAJOR", tablet_id:{id:455}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389074974000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305845058988660670-0](offset=1824702,size=1909,type=2,seq=0), type:"MINOR", tablet_id:{id:455}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178192070487000, upper_trans_version:1726178192070487000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337660633899, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcec60, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209461810}, this:0x7f92a3bcec60, timestamp:1727334209461810, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:55, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274239678928000, v:0}, rec_scn:{val:1726241818022255001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771548160-2](offset=1826816,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843202489049421-0](offset=1827149,size=180,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881173505-0](offset=1827329,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447693539-0](offset=1827555,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a794f0, 0x7f92a3bcec60, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.714446] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=66] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.714454] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.714522] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d34060, count:3, allocator:0x7f92864407d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287d34320, local_datums:0x7f9287d34320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:01:00.714539] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=15] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d38060, count:3, allocator:0x7f9286441558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287d38320, local_datums:0x7f9287d38320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:01:00.714788] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:00.714838] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=7][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.714843] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf2041c8 0xf217c49 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.714850] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:00.714853] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:00.714855] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9264a68108, hex: 313030303030333830, cstr: 100000380},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a680a8, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a680e0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-29413, column_count:20, rowkey_column_count:3, has_column_checksum:0, row_count:100, row_store_type:0, opt:0, var_column_count:0, row_offset:16082, original_length:16422, max_merged_trans_version:1726189672100745000, data_length:16454, data_zlength:5272, data_checksum:4171874375, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9287dce030, buf_size:5272, data_size:16454, row_count:100, column_count:20, max_merged_trans_version:1726189672100745000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:371002229, row_count_delta:22, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16422}) [2024-09-26 16:01:00.714867] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.714874] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=0] partition merge iter row count(i=0, row_count=23, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.714890] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=13] partition merge iter row count(i=1, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337660633318, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.714909] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660714469, end_time=1727337660714909, runtime=440, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FD-0-0, dag_ret:0, dag_status:2, start_time:1727337660633556, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.714919] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=81363, dag_cnt=34, dag_cnts_[dag.get_type()]=34, &dag=0x7f9264d76560, dag={ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C6FD-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660633556, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.714928] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C6FD-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660633541, task_id:YB42C0A8002D-0006230056D8C6FD-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.714966] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337660633814, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.714977] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753327496, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753327496, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.714990] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.714992] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.714989] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337660633847, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.715000] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.714998] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:444}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753327496, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.715014] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.715017] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.715016] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:444}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753327496, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:444}, table_count:1, [{i:0, table_key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8632d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660709261, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.74ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.715022] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:331}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.715039] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C707-0-0, dag_ret:0, dag_status:2, start_time:1727337660709242, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab490, output_block_cnt_arr:0x7f92684ab498, concurrent_cnt:1, estimate_row_cnt:4, estimate_occupy_size:6291456, latest_update_ts:1727337660709261, estimated_finish_time:1727337677715042}) [2024-09-26 16:01:00.715044] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d79d00, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C707-0-0, dag_ret:0, dag_status:2, start_time:1727337660709242, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.715049] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=5] task finish process(ret=0, start_time=1727337660709259, end_time=1727337660715049, runtime=5790, *this={this:0x7f9264d79d00, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C707-0-0, dag_ret:0, dag_status:2, start_time:1727337660709242, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.715040] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:331}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:331}, table_count:1, [{i:0, table_key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}, merge_scn:{val:1726241465393532000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8633ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660712494, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.52ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.715067] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C709-0-0, dag_ret:0, dag_status:2, start_time:1727337660712285, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a7480, output_block_cnt_arr:0x7f92a23a7488, concurrent_cnt:1, estimate_row_cnt:45, estimate_occupy_size:0, latest_update_ts:1727337660712494, estimated_finish_time:1727337677715066}) [2024-09-26 16:01:00.715075] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d7a760, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C709-0-0, dag_ret:0, dag_status:2, start_time:1727337660712285, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:01:00.715081] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=5] task finish process(ret=0, start_time=1727337660712493, end_time=1727337660715081, runtime=2588, *this={this:0x7f9264d7a760, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C709-0-0, dag_ret:0, dag_status:2, start_time:1727337660712285, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:01:00.715095] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=7] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.715099] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=3] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.715105] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660715103, task_id:YB42C0A8002D-0006230056D8C70D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.715108] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=3] schedule one task(task=0x7f9264d7bc20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.715126] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.715129] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.715131] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.715135] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.715167] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=331, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b56060, count:5, allocator:0x7f926912c7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9263b56360, local_datums:0x7f9263b56360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:01:00.715138] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8624c80, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:541, tablet_addr:[816017-1849-0-2305843684599267328-2](offset=1966080,size=629,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:492}, data_tablet_id:{id:492}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2528622955, row_count:340306}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8625434, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305850166778642432-0](offset=1949696,size=6666,type=2,seq=0), type:"MAJOR", tablet_id:{id:492}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389176297000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844149529467402-0](offset=1956362,size=1062,type=2,seq=0), type:"MINOR", tablet_id:{id:492}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178192725955000, upper_trans_version:1726178192725955000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337660633915, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd2310, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209496637}, this:0x7f92a3bd2310, timestamp:1727334209496637, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:62, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274246934057000, v:0}, rec_scn:{val:1726241820481609000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697941504-2](offset=1961984,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843215374086476-0](offset=1962316,size=192,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881308684-0](offset=1962508,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447828718-0](offset=1962734,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a78550, 0x7f92a3bd2310, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.715184] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=46] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.715192] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.715389] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.715428] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.715432] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.715442] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=0] partition merge iter row count(i=0, row_count=196, ghost_row_count=0, pkey={tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, table={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337660633847, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.715478] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660715118, end_time=1727337660715478, runtime=360, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C709-0-0, dag_ret:0, dag_status:2, start_time:1727337660712285, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:01:00.715488] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3203, dag_cnt=33, dag_cnts_[dag.get_type()]=33, &dag=0x7f9264d7a3a0, dag={ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C709-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660712285, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}) [2024-09-26 16:01:00.715496] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C709-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660712277, task_id:YB42C0A8002D-0006230056D8C709-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.715559] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=444, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a64060, count:6, allocator:0x7f92636447d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264a64390, local_datums:0x7f9264a64390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:01:00.715720] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.715751] ERROR issue_dba_error (ob_log.cpp:1875) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.715754] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.715762] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=0] partition merge iter row count(i=0, row_count=207, ghost_row_count=0, pkey={tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337660633814, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.715791] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660715526, end_time=1727337660715790, runtime=264, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C707-0-0, dag_ret:0, dag_status:2, start_time:1727337660709242, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.715800] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=6558, dag_cnt=32, dag_cnts_[dag.get_type()]=32, &dag=0x7f9264d79940, dag={ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C707-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660709242, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.715807] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C707-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660709232, task_id:YB42C0A8002D-0006230056D8C707-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.715861] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660715859, task_id:YB42C0A8002D-0006230056D8C70E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.715867] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=5] schedule one task(task=0x7f9264d7c150, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.715874] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337660715873, task_id:YB42C0A8002D-0006230056D8C70F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.715877] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=2] schedule one task(task=0x7f9264d7c680, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.715928] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.715934] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.715938] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.715943] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.715946] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b863fe60, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:511, tablet_addr:[816017-1849-0-2305843652386852864-2](offset=1806336,size=599,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:501}, data_tablet_id:{id:501}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1792252290, row_count:320564}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8640560, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305866008765333504-0](offset=1769472,size=21420,type=2,seq=0), type:"MAJOR", tablet_id:{id:501}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389518729000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844002426672044-0](offset=1790892,size=925,type=2,seq=0), type:"MINOR", tablet_id:{id:501}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178194873791000, upper_trans_version:1726178194873791000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337660633931, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd03d0, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495744}, this:0x7f92a3bd03d0, timestamp:1727334209495744, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:58, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274254042757000, v:0}, rec_scn:{val:1726241824983317000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771515392-2](offset=1794048,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161663821-0](offset=1794381,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881140719-0](offset=1794543,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843215373918929-0](offset=1794769,size=192,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a78d20, 0x7f92a3bd03d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.715985] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=39] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.715992] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.716011] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.716014] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.716015] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.716018] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.716020] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b862a0a0, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:490, tablet_addr:[816017-1849-0-2305843629838340096-2](offset=1871872,size=578,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:472}, data_tablet_id:{id:472}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:19976489, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862a734, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844179594129408-0](offset=1847296,size=1090,type=2,seq=0), type:"MAJOR", tablet_id:{id:472}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389523010000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843910084932674-0](offset=1848386,size=839,type=2,seq=0), type:"MINOR", tablet_id:{id:472}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178194887823000, upper_trans_version:1726178194887823000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337660633947, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd4080, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209505593}, this:0x7f92a3bd4080, timestamp:1727334209505593, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:65, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274254072414000, v:0}, rec_scn:{val:1726241829658678000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697839104-2](offset=1859584,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834376524-0](offset=1859916,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881206236-0](offset=1860060,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447726270-0](offset=1860286,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7ac60, 0x7f92a3bd4080, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.716049] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=29] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.716054] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.716138] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337660633882, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.716149] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753417144, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828753417144, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.716225] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:01:00.716229] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.716235] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:487}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753417144, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.716258] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:487}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753417144, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:487}, table_count:1, [{i:0, table_key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660712546, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.69ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.716347] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=20] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:01:00.716442] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70B-0-0, dag_ret:0, dag_status:2, start_time:1727337660712524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684b14a0, output_block_cnt_arr:0x7f92684b14a8, concurrent_cnt:1, estimate_row_cnt:2199, estimate_occupy_size:0, latest_update_ts:1727337660712546, estimated_finish_time:1727337677716441}) [2024-09-26 16:01:00.716449] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70B-0-0, dag_ret:0, dag_status:2, start_time:1727337660712524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.716455] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=6] task finish process(ret=0, start_time=1727337660712544, end_time=1727337660716455, runtime=3911, *this={this:0x7f9264d7b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70B-0-0, dag_ret:0, dag_status:2, start_time:1727337660712524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.716508] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=2] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.716556] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=487, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:11}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:8, allocator:0x7f92636447d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92665f83e0, local_datums:0x7f92665f83e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:01:00.716788] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.716826] ERROR issue_dba_error (ob_log.cpp:1875) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.716830] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.716839] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=0] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337660633882, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.716860] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660716525, end_time=1727337660716860, runtime=335, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70B-0-0, dag_ret:0, dag_status:2, start_time:1727337660712524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.716869] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4345, dag_cnt=31, dag_cnts_[dag.get_type()]=31, &dag=0x7f9264d7ae00, dag={ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660712524, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.716877] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70B-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660712519, task_id:YB42C0A8002D-0006230056D8C70B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.716903] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337660716902, task_id:YB42C0A8002D-0006230056D8C710-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.716909] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=6] schedule one task(task=0x7f9264d7cbb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.716926] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.716929] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.716931] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726077706339704027) [2024-09-26 16:01:00.716935] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.716938] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077600415496000, old_tablet_snaphsot_version=1726077706339704027, tablet={this:0x7f92b864a0a0, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[720905-1289-0-2305843650239553536-1](offset=1990656,size=597,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:342}, data_tablet_id:{id:342}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1725843677180998000, v:0}, clog_checkpoint_scn:{val:1725843677180998000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726077706339704027, multi_version_start:1726077600415496000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2338495768, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864a7a0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[720905-1289-0-2305844556477644800-0](offset=1982464,size=1441,type=2,seq=0), type:"MAJOR", tablet_id:{id:342}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725843393342764001}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337660633963, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[720905-1289-0-2305843251881332736-1](offset=1986560,size=226,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[720905-1289-0-2305843183161856226-0](offset=1986786,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[720905-1289-0-2305843251881333124-0](offset=1986948,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[720905-1289-0-2305843216447853158-0](offset=1987174,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7cba0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:01:00.716967] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=28] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.716972] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.717372] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337660633863, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.717383] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753357664, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753357664, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.717441] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:01:00.717445] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.717450] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:458}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753357664, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.717469] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:458}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753357664, storage_schema:{this:0x7f9287db9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:458}, table_count:1, [{i:0, table_key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8627b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660712382, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.07ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.717597] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=18] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:01:00.717752] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70A-0-0, dag_ret:0, dag_status:2, start_time:1727337660712296, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db94a0, output_block_cnt_arr:0x7f9287db94a8, concurrent_cnt:1, estimate_row_cnt:14598, estimate_occupy_size:4194304, latest_update_ts:1727337660712382, estimated_finish_time:1727337677717754}) [2024-09-26 16:01:00.717763] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=11] succeed to init merge ctx(task={this:0x7f9264d7ac90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70A-0-0, dag_ret:0, dag_status:2, start_time:1727337660712296, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.717770] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=6] task finish process(ret=0, start_time=1727337660712380, end_time=1727337660717769, runtime=5389, *this={this:0x7f9264d7ac90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70A-0-0, dag_ret:0, dag_status:2, start_time:1727337660712296, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.717795] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=3] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.717806] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=14] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337660633915, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.717817] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753437040, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=57, max_schema_version_in_memtable=1710828753437040, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.717839] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=458, range=0x7f92a98d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a60060, count:8, allocator:0x7f92a31fc7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264a603e0, local_datums:0x7f9264a603e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:01:00.717877] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=12] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1362, total_rows=1362) [2024-09-26 16:01:00.717880] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.717888] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:492}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:12, rowkey_column_count:11, schema_rowkey_col_cnt:9, full_stored_col_cnt:59, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753437040, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=28 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.717912] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=19] succeed to build merge ctx(tablet_id={id:492}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753437040, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:492}, table_count:1, [{i:0, table_key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8624c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660715120, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.77ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.717988] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=22] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70D-0-0, dag_ret:0, dag_status:2, start_time:1727337660715108, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f4c0, output_block_cnt_arr:0x7f9264b1f4c8, concurrent_cnt:1, estimate_row_cnt:1362, estimate_occupy_size:2097152, latest_update_ts:1727337660715120, estimated_finish_time:1727337677717989}) [2024-09-26 16:01:00.717996] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d7bc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70D-0-0, dag_ret:0, dag_status:2, start_time:1727337660715108, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.718002] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=5] task finish process(ret=0, start_time=1727337660715118, end_time=1727337660718002, runtime=2884, *this={this:0x7f9264d7bc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70D-0-0, dag_ret:0, dag_status:2, start_time:1727337660715108, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.718056] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.718104] ERROR issue_dba_error (ob_log.cpp:1875) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.718107] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.718118] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=0] partition merge iter row count(i=0, row_count=247, ghost_row_count=0, pkey={tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337660633863, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.718139] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660717808, end_time=1727337660718139, runtime=331, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70A-0-0, dag_ret:0, dag_status:2, start_time:1727337660712296, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.718149] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5852, dag_cnt=30, dag_cnts_[dag.get_type()]=30, &dag=0x7f9264d7a8d0, dag={ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660712296, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.718156] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70A-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660712293, task_id:YB42C0A8002D-0006230056D8C70A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.718175] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=5] schedule one task(task=0x7f9264bac030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.718185] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337660718184, task_id:YB42C0A8002D-0006230056D8C711-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.718188] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=3] schedule one task(task=0x7f9264d7d0e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.718235] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=492, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:57, schema_rowkey_cnt:9, rowkey_cnt:11, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:9, column_cnt:59}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:11, allocator:0x7f926912a7d8, capacity:11, init_cnt:11}, datum_utils:{is_oracle_mode:false, rowkey_cnt:11, col_cnt:11, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:59, datum_buffer:{capacity:59, datums:0x7f92665fe050, local_datums:0x7f92684b4440}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,col_id=43:NOP,col_id=44:NOP,col_id=45:NOP,col_id=46:NOP,col_id=47:NOP,col_id=48:NOP,col_id=49:NOP,col_id=50:NOP,col_id=51:NOP,col_id=52:NOP,col_id=53:NOP,col_id=54:NOP,col_id=55:NOP,col_id=56:NOP,col_id=57:NOP,col_id=58:NOP,]}) [2024-09-26 16:01:00.718432] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.718470] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.718474] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.718485] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=0] partition merge iter row count(i=0, row_count=61, ghost_row_count=0, pkey={tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337660633915, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.718504] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660718206, end_time=1727337660718504, runtime=298, *this={this:0x7f9264bac030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70D-0-0, dag_ret:0, dag_status:2, start_time:1727337660715108, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.718516] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3408, dag_cnt=29, dag_cnts_[dag.get_type()]=29, &dag=0x7f9264d7b860, dag={ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660715108, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.718525] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70D-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660715103, task_id:YB42C0A8002D-0006230056D8C70D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.718560] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.718566] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.718568] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.718572] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.718576] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.718582] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660718581, task_id:YB42C0A8002D-0006230056D8C712-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.718584] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.718585] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=3] schedule one task(task=0x7f9264d7d610, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.718629] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.718632] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.718634] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.718637] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.718640] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8641da0, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:487, tablet_addr:[816017-1849-0-2305843626617249792-2](offset=2007040,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101009}, data_tablet_id:{id:140}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726073917016589000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3251381279, row_count:150}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8642458, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844188184215552-0](offset=1998848,size=1098,type=2,seq=0), type:"MAJOR", tablet_id:{id:101009}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726070403667750001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843957329724490-0](offset=1999946,size=883,type=2,seq=0), type:"MINOR", tablet_id:{id:101009}, scn_range:{start_scn:{val:1726073917016589000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726167395874925000, upper_trans_version:1726167395874925000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337660633995, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd0ba0, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495816}, this:0x7f92a3bd0ba0, timestamp:1727334209495816, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:59, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726256203821911000, v:0}, rec_scn:{val:1726241803398655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697982464-2](offset=2002944,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276970828-0](offset=2003276,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881349602-0](offset=2003426,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7bc00, 0x7f92a3bd0ba0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.718679] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=38] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.718684] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.718882] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337660633931, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.718892] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753441792, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828753441792, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.718948] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1295, total_rows=1295) [2024-09-26 16:01:00.718951] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.718957] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:501}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753441792, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.719010] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=48] succeed to build merge ctx(tablet_id={id:501}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753441792, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:501}, table_count:1, [{i:0, table_key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660715922, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.03ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.719096] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70E-0-0, dag_ret:0, dag_status:2, start_time:1727337660715866, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b490, output_block_cnt_arr:0x7f9263a4b498, concurrent_cnt:1, estimate_row_cnt:1295, estimate_occupy_size:0, latest_update_ts:1727337660715922, estimated_finish_time:1727337677719095}) [2024-09-26 16:01:00.719104] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d7c150, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70E-0-0, dag_ret:0, dag_status:2, start_time:1727337660715866, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.719110] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=6] task finish process(ret=0, start_time=1727337660715920, end_time=1727337660719110, runtime=3190, *this={this:0x7f9264d7c150, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70E-0-0, dag_ret:0, dag_status:2, start_time:1727337660715866, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.719135] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=2] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.719157] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337660633947, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.719167] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753360568, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753360568, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.719176] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.719179] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.719184] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:472}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753360568, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.719197] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:472}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753360568, storage_schema:{this:0x7f9287db7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:472}, table_count:1, [{i:0, table_key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660716008, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.17ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.719215] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70F-0-0, dag_ret:0, dag_status:2, start_time:1727337660715876, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db7460, output_block_cnt_arr:0x7f9287db7468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337660716008, estimated_finish_time:1727337677719214}) [2024-09-26 16:01:00.719223] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d7c680, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70F-0-0, dag_ret:0, dag_status:2, start_time:1727337660715876, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.719228] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=4] task finish process(ret=0, start_time=1727337660716007, end_time=1727337660719228, runtime=3221, *this={this:0x7f9264d7c680, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70F-0-0, dag_ret:0, dag_status:2, start_time:1727337660715876, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.719245] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=8] schedule one task(task=0x7f9264d852e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.719280] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=501, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:6, allocator:0x7f92691287d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5e390, local_datums:0x7f9263b5e390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:00.719303] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=472, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b0060, count:3, allocator:0x7f926912a7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f92684b0320, local_datums:0x7f92684b0320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:01:00.719340] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.719364] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=24][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.719374] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.719377] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.719387] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=0] partition merge iter row count(i=0, row_count=17, ghost_row_count=0, pkey={tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337660633947, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.719400] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.719403] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.719406] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660719261, end_time=1727337660719405, runtime=144, *this={this:0x7f9264d852e0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70F-0-0, dag_ret:0, dag_status:2, start_time:1727337660715876, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.719415] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3538, dag_cnt=28, dag_cnts_[dag.get_type()]=28, &dag=0x7f9264d7c2c0, dag={ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660715876, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.719413] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=0] partition merge iter row count(i=0, row_count=56, ghost_row_count=0, pkey={tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337660633931, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.719422] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70F-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660715873, task_id:YB42C0A8002D-0006230056D8C70F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.719431] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660719247, end_time=1727337660719431, runtime=184, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70E-0-0, dag_ret:0, dag_status:2, start_time:1727337660715866, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.719442] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] succeed to add sys task(task={start_time:1727337660719440, task_id:YB42C0A8002D-0006230056D8C713-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.719445] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=2] schedule one task(task=0x7f9264d7db40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.719448] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337660633979, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.719481] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=33] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.719491] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.719494] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.719499] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:140}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.719513] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:140}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:140}, table_count:1, [{i:0, table_key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8622d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660718555, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=942us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.719532] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C711-0-0, dag_ret:0, dag_status:2, start_time:1727337660718188, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a7460, output_block_cnt_arr:0x7f92a23a7468, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337660718555, estimated_finish_time:1727337677719532}) [2024-09-26 16:01:00.719538] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C711-0-0, dag_ret:0, dag_status:2, start_time:1727337660718188, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.719544] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=5] task finish process(ret=0, start_time=1727337660718553, end_time=1727337660719543, runtime=990, *this={this:0x7f9264d7d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C711-0-0, dag_ret:0, dag_status:2, start_time:1727337660718188, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.719560] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=2] schedule one task(task=0x7f9264afa030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.719577] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.719581] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.719583] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:00.719587] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.719590] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.719596] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.719637] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=140, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:3, allocator:0x7f92636167d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263a38320, local_datums:0x7f9263a38320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:01:00.719697] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.719729] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.719732] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.719740] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337660633979, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.719757] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660719610, end_time=1727337660719757, runtime=147, *this={this:0x7f9264afa030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C711-0-0, dag_ret:0, dag_status:2, start_time:1727337660718188, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.719766] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1577, dag_cnt=27, dag_cnts_[dag.get_type()]=27, &dag=0x7f9264d7cd20, dag={ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C711-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660718188, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.719811] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C711-0-0] [lt=43] succeed to del sys task(removed_task={start_time:1727337660718184, task_id:YB42C0A8002D-0006230056D8C711-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.719827] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337660719825, task_id:YB42C0A8002D-0006230056D8C714-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.719833] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=5] schedule one task(task=0x7f9264d7e070, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.719843] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337660633899, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.719852] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753332736, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=36, max_schema_version_in_memtable=1710828753332736, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.719923] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.719928] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.719931] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:00.719936] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.719985] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:01:00.719940] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726112893187579000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864b040, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[812168-2208-0-2305843644869742592-2](offset=888832,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:323}, data_tablet_id:{id:323}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726112893187579000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2130074741, row_count:1002}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864b71c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[812168-2208-0-2305844507084419072-0](offset=880640,size=1395,type=2,seq=0), type:"MAJOR", tablet_id:{id:323}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562725180000}}, minor_tables:{{i:0, addr:[812168-2208-0-2305843933706286451-0](offset=882035,size=861,type=2,seq=0), type:"MINI", tablet_id:{id:323}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726112893187579000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726112601715820000, upper_trans_version:1726112601715820000}, {i:1, addr:[812168-2208-0-2305843933706287312-0](offset=882896,size=861,type=2,seq=0), type:"MINI", tablet_id:{id:323}, scn_range:{start_scn:{val:1726112893187579000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149985954467000, upper_trans_version:1726149985954467000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337660634026, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd9e40, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220581091}, this:0x7f92a3bd9e40, timestamp:1727334220581091, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:77, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265243868557000, v:0}, rec_scn:{val:1726264808086409001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[812168-2208-0-2305843482734723072-2](offset=884736,size=441,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[812168-2208-0-2305843176718303673-0](offset=885177,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843251880231509-0](offset=885333,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[812168-2208-0-2305843216446751543-0](offset=885559,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7d370, 0x7f92a3bd9e40, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.719988] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.719988] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=48] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.719995] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.719994] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:455}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:10, rowkey_column_count:9, schema_rowkey_col_cnt:7, full_stored_col_cnt:38, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753332736, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=26 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.720027] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=28] succeed to build merge ctx(tablet_id={id:455}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753332736, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:455}, table_count:1, [{i:0, table_key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660714353, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.64ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.720180] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=29] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:01:00.720337] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70C-0-0, dag_ret:0, dag_status:2, start_time:1727337660714023, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db54b0, output_block_cnt_arr:0x7f9287db54b8, concurrent_cnt:1, estimate_row_cnt:3824, estimate_occupy_size:4194304, latest_update_ts:1727337660714353, estimated_finish_time:1727337677720339}) [2024-09-26 16:01:00.720367] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=30] succeed to init merge ctx(task={this:0x7f9264d7b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70C-0-0, dag_ret:0, dag_status:2, start_time:1727337660714023, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.720374] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=7] task finish process(ret=0, start_time=1727337660714351, end_time=1727337660720374, runtime=6023, *this={this:0x7f9264d7b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70C-0-0, dag_ret:0, dag_status:2, start_time:1727337660714023, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.720393] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4527, dag_cnt=26, dag_cnts_[dag.get_type()]=26, &dag=0x7f9264d7bd90, dag={ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660715866, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.720405] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70E-0-0] [lt=9] succeed to del sys task(removed_task={start_time:1727337660715859, task_id:YB42C0A8002D-0006230056D8C70E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.720422] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337660633963, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.720432] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753001832, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=21, max_schema_version_in_memtable=1710828753001832, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.720446] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.720449] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.720455] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:342}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:23, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753001832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.720473] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:342}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753001832, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:342}, table_count:1, [{i:0, table_key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660716922, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.53ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.720512] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=33] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C710-0-0, dag_ret:0, dag_status:2, start_time:1727337660716909, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d490, output_block_cnt_arr:0x7f9264b1d498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337660716922, estimated_finish_time:1727337677720512}) [2024-09-26 16:01:00.720519] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C710-0-0, dag_ret:0, dag_status:2, start_time:1727337660716909, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.720524] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=4] task finish process(ret=0, start_time=1727337660716920, end_time=1727337660720524, runtime=3604, *this={this:0x7f9264d7cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C710-0-0, dag_ret:0, dag_status:2, start_time:1727337660716909, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.720542] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=2] schedule one task(task=0x7f9264afa030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.720546] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=3] schedule one task(task=0x7f9264b36030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.720550] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660720549, task_id:YB42C0A8002D-0006230056D8C715-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.720556] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=5] schedule one task(task=0x7f9264d7e5a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.720583] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=455, range=0x7f92a9952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:36, schema_rowkey_cnt:7, rowkey_cnt:9, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:7, column_cnt:38}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685e8060, count:9, allocator:0x7f92b321a7d8, capacity:9, init_cnt:9}, datum_utils:{is_oracle_mode:false, rowkey_cnt:9, col_cnt:9, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:38, datum_buffer:{capacity:38, datums:0x7f92685e9340, local_datums:0x7f92685e83f0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,]}) [2024-09-26 16:01:00.720596] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=342, range=0x7f92a9ed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:21, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:23}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d30060, count:6, allocator:0x7f92bc8f47d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:23, datum_buffer:{capacity:32, datums:0x7f9287d30390, local_datums:0x7f9287d30390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,]}) [2024-09-26 16:01:00.720631] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=17][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:00.720676] ERROR issue_dba_error (ob_log.cpp:1875) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.720679] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.720684] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=5][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:00.720686] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:00.720688] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d380a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 12, flag: 0, null: 0, ptr: 0x7f9287d381b0, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38118, hex: 420B000000000000, int: 2882},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38150, hex: 98A4BFDF9C5C0BE8, int: -1726184203327921000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38188, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:18768, column_count:23, rowkey_column_count:6, has_column_checksum:0, row_count:3, row_store_type:0, opt:4, var_column_count:0, row_offset:494, original_length:446, max_merged_trans_version:1726184204644606000, data_length:446, data_zlength:446, data_checksum:3585221911, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92b6c04090, buf_size:446, data_size:446, row_count:3, column_count:23, max_merged_trans_version:1726184204644606000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1373226696, row_count_delta:1, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:446}) [2024-09-26 16:01:00.720700] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:00.720702] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=2][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:01:00.720704] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=2][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:01:00.720705] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=1][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:01:00.720707] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=2][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:01:00.720709] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=1] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337660633963, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.720723] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=11][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:00.720732] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:00.720736] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:00.720738] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337660720559, end_time=1727337660720738, runtime=179, *this={this:0x7f9264b36030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C710-0-0, dag_ret:0, dag_status:2, start_time:1727337660716909, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.720744] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264b36030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C710-0-0, dag_ret:0, dag_status:2, start_time:1727337660716909, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:00.720752] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=3842, dag_cnt=25, dag_cnts_[dag.get_type()]=25, &dag=0x7f9264d7c7f0, dag={ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C710-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660716909, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.720770] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.720773] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C710-0-0] [lt=18] succeed to del sys task(removed_task={start_time:1727337660716902, task_id:YB42C0A8002D-0006230056D8C710-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.720775] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.720777] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:00.720782] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.720785] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.720789] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660720788, task_id:YB42C0A8002D-0006230056D8C716-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.720791] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.720792] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=3] schedule one task(task=0x7f9264d7ead0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.720809] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.720812] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.720814] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:00.720818] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.720848] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.720822] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864eec0, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[814042-5586-0-2305843650238771200-2](offset=1208320,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:297}, data_tablet_id:{id:297}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726063409221604000, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2361333062, row_count:24}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844531780800512-0](offset=1200128,size=1418,type=2,seq=0), type:"MAJOR", tablet_id:{id:297}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063202518238000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843919747962250-0](offset=1201546,size=848,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726063409221604000, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557514739000, upper_trans_version:1726121557514739000}, {i:1, addr:[814042-5586-0-2305843920821704922-0](offset=1202394,size=849,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149603136610000, upper_trans_version:1726149603136610000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337660634059, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd86d0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220478269}, this:0x7f92a3bd86d0, timestamp:1727334220478269, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:74, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726264804632655000, v:0}, rec_scn:{val:1726264804632655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513817088-2](offset=1204224,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843183161074102-0](offset=1204662,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880551000-0](offset=1204824,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447071034-0](offset=1205050,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7eae0, 0x7f92a3bd86d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.720862] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=39] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.720881] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=14] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.720888] ERROR issue_dba_error (ob_log.cpp:1875) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.720891] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.720901] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=0] partition merge iter row count(i=0, row_count=140, ghost_row_count=0, pkey={tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337660633899, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.720921] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660720554, end_time=1727337660720920, runtime=366, *this={this:0x7f9264afa030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70C-0-0, dag_ret:0, dag_status:2, start_time:1727337660714023, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.720929] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=6906, dag_cnt=24, dag_cnts_[dag.get_type()]=24, &dag=0x7f9264d7b330, dag={ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C70C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660714023, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.720936] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C70C-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660714016, task_id:YB42C0A8002D-0006230056D8C70C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.720948] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660720947, task_id:YB42C0A8002D-0006230056D8C717-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.720951] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=3] schedule one task(task=0x7f9264d7f000, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.720994] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.720998] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.721000] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:00.721004] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.721007] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.721022] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=10] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.723104] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337660634010, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.723121] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.723135] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.723140] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.723146] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101010}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:01:00.723163] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101010}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101010}, table_count:1, [{i:0, table_key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8642d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660719565, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.58ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.723190] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C713-0-0, dag_ret:0, dag_status:2, start_time:1727337660719444, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3470, output_block_cnt_arr:0x7f92728f3478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337660719565, estimated_finish_time:1727337677723190}) [2024-09-26 16:01:00.723197] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7db40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C713-0-0, dag_ret:0, dag_status:2, start_time:1727337660719444, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.723202] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=5] task finish process(ret=0, start_time=1727337660719563, end_time=1727337660723202, runtime=3639, *this={this:0x7f9264d7db40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C713-0-0, dag_ret:0, dag_status:2, start_time:1727337660719444, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.723221] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.723274] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101010, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b2060, count:4, allocator:0x7f92864407d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684b2350, local_datums:0x7f92684b2350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:00.723329] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=30][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.723371] ERROR issue_dba_error (ob_log.cpp:1875) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.723374] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.723384] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337660634010, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.723406] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660723241, end_time=1727337660723406, runtime=165, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C713-0-0, dag_ret:0, dag_status:2, start_time:1727337660719444, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.723416] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3972, dag_cnt=23, dag_cnts_[dag.get_type()]=23, &dag=0x7f9264d7d780, dag={ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C713-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660719444, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.723425] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C713-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660719440, task_id:YB42C0A8002D-0006230056D8C713-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.723454] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] succeed to add sys task(task={start_time:1727337660723452, task_id:YB42C0A8002D-0006230056D8C718-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.723460] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=6] schedule one task(task=0x7f9264d7f530, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.723507] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.723511] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.723513] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:00.723517] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.723521] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.723528] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.723982] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337660633995, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.723996] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.724006] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.724009] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.724014] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101009}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:01:00.724029] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:101009}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101009}, table_count:1, [{i:0, table_key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8641da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660718625, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.39ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.724050] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C712-0-0, dag_ret:0, dag_status:2, start_time:1727337660718585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a43470, output_block_cnt_arr:0x7f9263a43478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337660718625, estimated_finish_time:1727337677724050}) [2024-09-26 16:01:00.724056] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7d610, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C712-0-0, dag_ret:0, dag_status:2, start_time:1727337660718585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.724061] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=4] task finish process(ret=0, start_time=1727337660718623, end_time=1727337660724060, runtime=5437, *this={this:0x7f9264d7d610, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C712-0-0, dag_ret:0, dag_status:2, start_time:1727337660718585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.724101] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=4] schedule one task(task=0x7f9264c3a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.724154] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101009, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:4, allocator:0x7f92864407d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263a3e350, local_datums:0x7f9263a3e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:00.724190] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.724227] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.724230] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.724240] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337660633995, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.724260] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660724123, end_time=1727337660724259, runtime=136, *this={this:0x7f9264c3a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C712-0-0, dag_ret:0, dag_status:2, start_time:1727337660718585, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.724269] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5684, dag_cnt=22, dag_cnts_[dag.get_type()]=22, &dag=0x7f9264d7d250, dag={ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C712-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660718585, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.724277] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C712-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660718581, task_id:YB42C0A8002D-0006230056D8C712-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.724300] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337660724298, task_id:YB42C0A8002D-0006230056D8C719-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.724306] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=5] schedule one task(task=0x7f9264d7fa60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.724365] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.724370] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.724372] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:00.724376] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.724379] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.724385] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.725993] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337660634059, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.726004] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=16, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.726017] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.726022] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.726028] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:297}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:18, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.726018] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337660634042, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.726044] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=25] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=41, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.726060] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.726045] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:297}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:297}, table_count:1, [{i:0, table_key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660720804, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.22ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.726064] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.726082] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C716-0-0, dag_ret:0, dag_status:2, start_time:1727337660720792, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1490, output_block_cnt_arr:0x7f92728f1498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337660720804, estimated_finish_time:1727337677726081}) [2024-09-26 16:01:00.726070] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:377}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:43, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.726089] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7ead0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C716-0-0, dag_ret:0, dag_status:2, start_time:1727337660720792, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.726095] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=5] task finish process(ret=0, start_time=1727337660720802, end_time=1727337660726094, runtime=5292, *this={this:0x7f9264d7ead0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C716-0-0, dag_ret:0, dag_status:2, start_time:1727337660720792, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.726115] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=2] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.726100] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=24] succeed to build merge ctx(tablet_id={id:377}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:377}, table_count:1, [{i:0, table_key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660720764, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.30ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.726146] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=40] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C715-0-0, dag_ret:0, dag_status:2, start_time:1727337660720555, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b21480, output_block_cnt_arr:0x7f9264b21488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337660720764, estimated_finish_time:1727337677726146}) [2024-09-26 16:01:00.726153] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C715-0-0, dag_ret:0, dag_status:2, start_time:1727337660720555, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.726159] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=5] task finish process(ret=0, start_time=1727337660720761, end_time=1727337660726159, runtime=5398, *this={this:0x7f9264d7e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C715-0-0, dag_ret:0, dag_status:2, start_time:1727337660720555, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.726168] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=5] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.726225] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=377, range=0x7f92aa2d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:41, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:43}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a36060, count:5, allocator:0x7f92bc8f47d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:43, datum_buffer:{capacity:43, datums:0x7f9263a372b0, local_datums:0x7f9263a36360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,]}) [2024-09-26 16:01:00.726265] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.726306] ERROR issue_dba_error (ob_log.cpp:1875) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.726309] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.726318] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=0] partition merge iter row count(i=0, row_count=6, ghost_row_count=0, pkey={tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337660634042, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.726339] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660726193, end_time=1727337660726339, runtime=146, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C715-0-0, dag_ret:0, dag_status:2, start_time:1727337660720555, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.726348] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5793, dag_cnt=21, dag_cnts_[dag.get_type()]=21, &dag=0x7f9264d7e1e0, dag={ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C715-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660720555, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.726356] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C715-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660720549, task_id:YB42C0A8002D-0006230056D8C715-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.726384] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337660634026, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.726392] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660726390, task_id:YB42C0A8002D-0006230056D8C71A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.726397] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=4] schedule one task(task=0x7f9264d7ff90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.726392] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752949376, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=20, max_schema_version_in_memtable=1710828752949376, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.726412] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.726415] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.726434] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.726439] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.726441] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:00.726419] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:323}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:22, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752949376, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.726446] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.726450] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=26] succeed to build merge ctx(tablet_id={id:323}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752949376, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:323}, table_count:1, [{i:0, table_key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660719916, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=6.50ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.726495] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=31] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C714-0-0, dag_ret:0, dag_status:2, start_time:1727337660719832, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a7480, output_block_cnt_arr:0x7f92a23a7488, concurrent_cnt:1, estimate_row_cnt:205, estimate_occupy_size:0, latest_update_ts:1727337660719916, estimated_finish_time:1727337677726494}) [2024-09-26 16:01:00.726449] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706339704027, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8654c80, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:522, tablet_addr:[809012-1445-0-2305843664196722688-2](offset=516096,size=610,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:334}, data_tablet_id:{id:334}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706339704027, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2773161616, row_count:13383}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86553c8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844791625629696-0](offset=507904,size=1660,type=2,seq=0), type:"MAJOR", tablet_id:{id:334}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843977729328764-0](offset=509564,size=902,type=2,seq=0), type:"MINI", tablet_id:{id:334}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337660634134, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bddcc0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221095878}, this:0x7f92a3bddcc0, timestamp:1727334221095878, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:85, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265286142368001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843367843975168-2](offset=512000,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843196045283662-0](offset=512334,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879858684-0](offset=512508,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843215372636894-0](offset=512734,size=192,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc57f0, 0x7f92a3bddcc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.726501] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7e070, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C714-0-0, dag_ret:0, dag_status:2, start_time:1727337660719832, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.726504] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=55] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.726506] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=4] task finish process(ret=0, start_time=1727337660719914, end_time=1727337660726506, runtime=6592, *this={this:0x7f9264d7e070, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C714-0-0, dag_ret:0, dag_status:2, start_time:1727337660719832, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.726512] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.726546] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=2] schedule one task(task=0x7f927282a3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.726586] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=323, range=0x7f92a9a56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:20, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:22}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:5, allocator:0x7f926363e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f92665f8360, local_datums:0x7f92665f8360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,]}) [2024-09-26 16:01:00.726596] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=297, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:16, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:18}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:6, allocator:0x7f92b321a7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:18, datum_buffer:{capacity:32, datums:0x7f9263a3e390, local_datums:0x7f9263a3e390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,]}) [2024-09-26 16:01:00.726636] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=25][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.726669] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.726671] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.726679] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337660634059, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.726697] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660726569, end_time=1727337660726697, runtime=128, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C716-0-0, dag_ret:0, dag_status:2, start_time:1727337660720792, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.726705] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5913, dag_cnt=20, dag_cnts_[dag.get_type()]=20, &dag=0x7f9264d7e710, dag={ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C716-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660720792, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.726712] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C716-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660720788, task_id:YB42C0A8002D-0006230056D8C716-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.726762] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337660634075, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.726774] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.726785] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.726788] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.726792] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101065}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.726808] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101065}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101065}, table_count:1, [{i:0, table_key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660720989, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.80ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.726829] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C717-0-0, dag_ret:0, dag_status:2, start_time:1727337660720951, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab490, output_block_cnt_arr:0x7f92684ab498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337660720989, estimated_finish_time:1727337677726829}) [2024-09-26 16:01:00.726835] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7f000, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C717-0-0, dag_ret:0, dag_status:2, start_time:1727337660720951, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.726839] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=4] task finish process(ret=0, start_time=1727337660720988, end_time=1727337660726839, runtime=5851, *this={this:0x7f9264d7f000, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C717-0-0, dag_ret:0, dag_status:2, start_time:1727337660720951, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.726842] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=18][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.726886] ERROR issue_dba_error (ob_log.cpp:1875) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.726889] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.726899] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=0] partition merge iter row count(i=0, row_count=171, ghost_row_count=0, pkey={tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337660634026, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.726920] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660726557, end_time=1727337660726920, runtime=363, *this={this:0x7f927282a3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C714-0-0, dag_ret:0, dag_status:2, start_time:1727337660719832, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.726932] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=7100, dag_cnt=19, dag_cnts_[dag.get_type()]=19, &dag=0x7f9264d7dcb0, dag={ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C714-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660719832, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.726939] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C714-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660719825, task_id:YB42C0A8002D-0006230056D8C714-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.726966] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=2] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.726972] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337660726971, task_id:YB42C0A8002D-0006230056D8C71B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.726975] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=3] schedule one task(task=0x7f9264d804c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.726979] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660726978, task_id:YB42C0A8002D-0006230056D8C71C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.726981] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=1] schedule one task(task=0x7f9264d809f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.727000] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.727004] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.727006] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:00.727010] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.727013] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.727019] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.727040] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.727022] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101065, range=0x7f92aa352200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d2e060, count:6, allocator:0x7f92b321a7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9287d2e390, local_datums:0x7f9287d2e390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:01:00.727043] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.727045] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:00.727047] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.727063] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=22][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.727050] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864cf80, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[814042-5586-0-2305843644870074368-2](offset=1220608,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:378}, data_tablet_id:{id:378}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2166189977, row_count:350}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864d65c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844352465928192-0](offset=1212416,size=1251,type=2,seq=0), type:"MAJOR", tablet_id:{id:378}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563080818000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843914379265251-0](offset=1213667,size=843,type=2,seq=0), type:"MINI", tablet_id:{id:378}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557779816000, upper_trans_version:1726121557779816000}, {i:1, addr:[814042-5586-0-2305843914379266094-0](offset=1214510,size=843,type=2,seq=0), type:"MINI", tablet_id:{id:378}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986451466000, upper_trans_version:1726149986451466000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337660634169, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdf430, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221743022}, this:0x7f92a3bdf430, timestamp:1727334221743022, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:88, clock:641728512}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293334630000, v:0}, rec_scn:{val:1726265293334630000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513829376-2](offset=1216512,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843176718635446-0](offset=1216950,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880563282-0](offset=1217106,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447083316-0](offset=1217332,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7f2b0, 0x7f92a3bdf430, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.727089] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=39] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.727094] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.727101] ERROR issue_dba_error (ob_log.cpp:1875) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.727104] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.727112] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=0] partition merge iter row count(i=0, row_count=15, ghost_row_count=0, pkey={tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337660634075, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.727131] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660726992, end_time=1727337660727131, runtime=139, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C717-0-0, dag_ret:0, dag_status:2, start_time:1727337660720951, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.727139] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=6188, dag_cnt=18, dag_cnts_[dag.get_type()]=18, &dag=0x7f9264d7ec40, dag={ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C717-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660720951, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.727145] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C717-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660720947, task_id:YB42C0A8002D-0006230056D8C717-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.727183] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337660727181, task_id:YB42C0A8002D-0006230056D8C71D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.727186] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=2] schedule one task(task=0x7f9264d80f20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.727200] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.727202] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.727204] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:00.727207] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.727209] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:2, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337660634194, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.727239] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=30] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.727243] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241548806527000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.728239] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337660634118, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.728253] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.728261] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337660634100, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.728281] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.728287] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.728293] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101066}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.728277] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.728327] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=38] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.728331] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.728315] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:101066}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101066}, table_count:1, [{i:0, table_key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8653ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660724359, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.93ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.728347] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C719-0-0, dag_ret:0, dag_status:2, start_time:1727337660724306, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3490, output_block_cnt_arr:0x7f92728f3498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337660724359, estimated_finish_time:1727337677728347}) [2024-09-26 16:01:00.728339] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:333}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.728354] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7fa60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C719-0-0, dag_ret:0, dag_status:2, start_time:1727337660724306, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.728360] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=6] task finish process(ret=0, start_time=1727337660724357, end_time=1727337660728360, runtime=4003, *this={this:0x7f9264d7fa60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C719-0-0, dag_ret:0, dag_status:2, start_time:1727337660724306, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.728361] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:333}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:333}, table_count:1, [{i:0, table_key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8651da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660723500, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.84ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.728384] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=2] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.728405] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=22] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C718-0-0, dag_ret:0, dag_status:2, start_time:1727337660723460, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad490, output_block_cnt_arr:0x7f92684ad498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337660723500, estimated_finish_time:1727337677728405}) [2024-09-26 16:01:00.728416] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=10] succeed to init merge ctx(task={this:0x7f9264d7f530, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C718-0-0, dag_ret:0, dag_status:2, start_time:1727337660723460, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.728421] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=5] task finish process(ret=0, start_time=1727337660723498, end_time=1727337660728421, runtime=4923, *this={this:0x7f9264d7f530, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C718-0-0, dag_ret:0, dag_status:2, start_time:1727337660723460, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.728430] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=6] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.728447] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337660634150, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.728472] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=24] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=14, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.728481] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.728484] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.728489] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:451}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:16, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.728488] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=333, range=0x7f92aa6d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fc060, count:7, allocator:0x7f92a33547d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f92665fc3b0, local_datums:0x7f92665fc3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:01:00.728505] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:451}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:451}, table_count:1, [{i:0, table_key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8650e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660726994, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.49ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.728523] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71B-0-0, dag_ret:0, dag_status:2, start_time:1727337660726975, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b470, output_block_cnt_arr:0x7f9263a4b478, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337660726994, estimated_finish_time:1727337677728523}) [2024-09-26 16:01:00.728528] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d804c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71B-0-0, dag_ret:0, dag_status:2, start_time:1727337660726975, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.728533] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=5] task finish process(ret=0, start_time=1727337660726993, end_time=1727337660728533, runtime=1540, *this={this:0x7f9264d804c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71B-0-0, dag_ret:0, dag_status:2, start_time:1727337660726975, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.728554] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337660634169, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.728562] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f92664ef330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.728570] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.728572] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.728576] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92664ef330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:378}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.728589] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:378}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f92664ef330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:378}, table_count:1, [{i:0, table_key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660727037, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.54ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.728604] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71C-0-0, dag_ret:0, dag_status:2, start_time:1727337660726980, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92664ef480, output_block_cnt_arr:0x7f92664ef488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337660727037, estimated_finish_time:1727337677728604}) [2024-09-26 16:01:00.728609] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d809f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71C-0-0, dag_ret:0, dag_status:2, start_time:1727337660726980, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.728614] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=4] task finish process(ret=0, start_time=1727337660727036, end_time=1727337660728613, runtime=1577, *this={this:0x7f9264d809f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71C-0-0, dag_ret:0, dag_status:2, start_time:1727337660726980, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.728627] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752511928, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.728634] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.728636] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.728639] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:121}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752511928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.728655] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=18][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.728651] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:121}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660727197, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.44ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.728665] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=12] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71D-0-0, dag_ret:0, dag_status:2, start_time:1727337660727185, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1480, output_block_cnt_arr:0x7f92728f1488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337660727197, estimated_finish_time:1727337677728665}) [2024-09-26 16:01:00.728670] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d80f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71D-0-0, dag_ret:0, dag_status:2, start_time:1727337660727185, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:00.728674] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=4] task finish process(ret=0, start_time=1727337660727196, end_time=1727337660728674, runtime=1478, *this={this:0x7f9264d80f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71D-0-0, dag_ret:0, dag_status:2, start_time:1727337660727185, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:00.728693] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337660634134, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.728700] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=19, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.728720] ERROR issue_dba_error (ob_log.cpp:1875) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.728723] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.728734] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.728737] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.728735] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=0] partition merge iter row count(i=0, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337660634100, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.728741] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:334}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:21, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.728758] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660728442, end_time=1727337660728758, runtime=316, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C718-0-0, dag_ret:0, dag_status:2, start_time:1727337660723460, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.728757] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:334}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92664ed330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:334}, table_count:1, [{i:0, table_key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8654c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660726419, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.32ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.728769] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5309, dag_cnt=17, dag_cnts_[dag.get_type()]=17, &dag=0x7f9264d7f170, dag={ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C718-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660723460, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.728778] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C718-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660723452, task_id:YB42C0A8002D-0006230056D8C718-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.728818] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71A-0-0, dag_ret:0, dag_status:2, start_time:1727337660726396, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92664ed4a0, output_block_cnt_arr:0x7f92664ed4a8, concurrent_cnt:1, estimate_row_cnt:993, estimate_occupy_size:0, latest_update_ts:1727337660726419, estimated_finish_time:1727337677728818}) [2024-09-26 16:01:00.728836] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=18] succeed to init merge ctx(task={this:0x7f9264d7ff90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71A-0-0, dag_ret:0, dag_status:2, start_time:1727337660726396, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.728843] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=7] task finish process(ret=0, start_time=1727337660726416, end_time=1727337660728843, runtime=2427, *this={this:0x7f9264d7ff90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71A-0-0, dag_ret:0, dag_status:2, start_time:1727337660726396, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.728893] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=3] schedule one task(task=0x7f9264d852e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.728901] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=6] schedule one task(task=0x7f9272926030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.728906] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=2] schedule one task(task=0x7f9262d14030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.728909] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=2] schedule one task(task=0x7f9264d84f50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.728914] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337660728912, task_id:YB42C0A8002D-0006230056D8C71E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.728917] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=3] schedule one task(task=0x7f9264d81450, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.728900] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101066, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:7}, cols_desc:{ObIArray:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665aa060, count:7, allocator:0x7f92636407d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f92665aa3b0, local_datums:0x7f92665aa3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:01:00.728936] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.728939] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.728941] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:00.728945] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.728949] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.728957] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.729048] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=23][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.729048] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=334, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:19, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:21}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684e6060, count:8, allocator:0x7f92ba7f67d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:21, datum_buffer:{capacity:32, datums:0x7f92684e63e0, local_datums:0x7f92684e63e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,]}) [2024-09-26 16:01:00.729091] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.729094] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.729103] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=0] partition merge iter row count(i=0, row_count=182, ghost_row_count=0, pkey={tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337660634118, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.729124] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660728869, end_time=1727337660729124, runtime=255, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C719-0-0, dag_ret:0, dag_status:2, start_time:1727337660724306, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.729133] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4827, dag_cnt=16, dag_cnts_[dag.get_type()]=16, &dag=0x7f9264d7f6a0, dag={ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C719-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660724306, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.729143] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C719-0-0] [lt=8] succeed to del sys task(removed_task={start_time:1727337660724298, task_id:YB42C0A8002D-0006230056D8C719-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.729197] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=121, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685ea060, count:5, allocator:0x7f928643e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92685ea360, local_datums:0x7f92685ea360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:01:00.729216] INFO [STORAGE] dump_info (ob_sstable_struct.cpp:262) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=15] dump merge info(msg="macro block builder close", output_row_per_s=0, new_macro_KB_per_s=0, *this={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:41, merge_start_time:1727337660729175, merge_finish_time:1727337660729216, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:0, max:0, sum:0, count:0}}, {type:"cost_time", info:{min:0, max:0, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:0, max:0, sum:0, count:0}}, {type:"incremental_row_count", info:{min:0, max:0, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:0, snapshot_version:0, start_scn:0, end_scn:0}, macro_id_list:"", comment:""}) [2024-09-26 16:01:00.729231] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=14] partition merge iter row count(i=0, row_count=0, ghost_row_count=0, pkey={tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, table={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337660634194, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.729243] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1544) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=10] merge macro blocks ok(idx_=0, task={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71D-0-0, dag_ret:0, dag_status:2, start_time:1727337660727185, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:00.729252] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=6] task finish process(ret=0, start_time=1727337660729172, end_time=1727337660729252, runtime=80, *this={this:0x7f9264d84f50, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71D-0-0, dag_ret:0, dag_status:2, start_time:1727337660727185, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:00.729276] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.729294] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=378, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d30060, count:5, allocator:0x7f928643e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9287d30360, local_datums:0x7f9287d30360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:01:00.729311] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.729314] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.729324] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=0] partition merge iter row count(i=0, row_count=153, ghost_row_count=0, pkey={tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337660634134, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.729342] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.729345] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.729345] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660729006, end_time=1727337660729345, runtime=339, *this={this:0x7f9264d852e0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71A-0-0, dag_ret:0, dag_status:2, start_time:1727337660726396, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.729355] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2959, dag_cnt=15, dag_cnts_[dag.get_type()]=15, &dag=0x7f9264d7fbd0, dag={ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660726396, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.729352] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337660634169, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.729362] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71A-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660726390, task_id:YB42C0A8002D-0006230056D8C71A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.729368] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660729268, end_time=1727337660729367, runtime=99, *this={this:0x7f9262d14030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71C-0-0, dag_ret:0, dag_status:2, start_time:1727337660726980, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.729375] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2395, dag_cnt=14, dag_cnts_[dag.get_type()]=14, &dag=0x7f9264d80630, dag={ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660726980, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.729381] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71C-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337660726978, task_id:YB42C0A8002D-0006230056D8C71C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.729443] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=2] schedule one task(task=0x7f9264d85170, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.729449] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337660729448, task_id:YB42C0A8002D-0006230056D8C71F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.729452] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=3] schedule one task(task=0x7f9264d81980, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.729456] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660729455, task_id:YB42C0A8002D-0006230056D8C720-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.729458] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=2] schedule one task(task=0x7f9264d81eb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.729484] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337660729483, task_id:YB42C0A8002D-0006230056D8C721-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.729487] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=3] schedule one task(task=0x7f9264d823e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.729524] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.729528] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.729530] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:00.729534] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.729537] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.729544] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.729566] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337660634217, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.729576] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752725472, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752725472, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.729588] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.729591] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.729585] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=451, range=0x7f92aae56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:14, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:16}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b56060, count:4, allocator:0x7f92636487d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:16, datum_buffer:{capacity:32, datums:0x7f9263b56350, local_datums:0x7f9263b56350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,]}) [2024-09-26 16:01:00.729595] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:215}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752725472, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.729610] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.729611] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:215}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752725472, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:215}, table_count:1, [{i:0, table_key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660728929, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=665us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.729632] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71E-0-0, dag_ret:0, dag_status:2, start_time:1727337660728917, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a7460, output_block_cnt_arr:0x7f92a23a7468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337660728929, estimated_finish_time:1727337677729632}) [2024-09-26 16:01:00.729638] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d81450, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71E-0-0, dag_ret:0, dag_status:2, start_time:1727337660728917, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.729642] ERROR issue_dba_error (ob_log.cpp:1875) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.729643] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=4] task finish process(ret=0, start_time=1727337660728927, end_time=1727337660729643, runtime=716, *this={this:0x7f9264d81450, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71E-0-0, dag_ret:0, dag_status:2, start_time:1727337660728917, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.729644] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.729650] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337660634150, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.729665] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.729668] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.729670] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:00.729672] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.729675] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.729671] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660729545, end_time=1727337660729671, runtime=126, *this={this:0x7f9272926030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71B-0-0, dag_ret:0, dag_status:2, start_time:1727337660726975, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.729679] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.729680] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2705, dag_cnt=13, dag_cnts_[dag.get_type()]=13, &dag=0x7f9264d80100, dag={ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660726975, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.729687] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660726971, task_id:YB42C0A8002D-0006230056D8C71B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.729692] INFO [STORAGE.COMPACTION] get_merged_sstable (ob_tablet_merge_task.cpp:1285) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=3] create new merged sstable(ctx.param_.tablet_id_={id:121}, snapshot_version=1726241132764173999, ctx.param_.merge_type_=3, ctx.create_snapshot_version_=0, table_mode_flag=0) [2024-09-26 16:01:00.729699] INFO [STORAGE] close (ob_index_block_builder.cpp:815) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=3] succeed to close sstable index builder(res={root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_block_ids_.count():0, other_block_ids_.count():0, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, data_column_cnt:10, data_column_checksums:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], row_count:0, max_merged_trans_version:0, contain_uncommitted_row:false, occupy_size:0, original_size:0, data_checksum:0, use_old_macro_block_count:0, compressor_type:1, root_row_store_type:0, nested_offset:0, nested_size:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}) [2024-09-26 16:01:00.729720] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.729723] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.729725] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:00.729712] INFO [STORAGE] init (ob_sstable.cpp:118) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=7] succeeded to init sstable(ret=0, this={ObITable:{this:0x7f92728f0128, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f92728f0128, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:null, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:01:00.729729] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.729733] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.729740] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.729730] INFO [STORAGE.COMPACTION] create_sstable (ob_tablet_merge_ctx.cpp:367) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=18] succeed to merge sstable(param={table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_logic_seq:0, schema_version:1710828752511928, create_snapshot_version:0, progressive_merge_round:0, progressive_merge_step:0, is_ready_for_read:true, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, index_type:0, root_block_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), root_block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, root_row_store_type:0, latest_row_store_type:1, data_index_tree_height:0, data_block_macro_meta_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), data_block_macro_meta:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, use_old_macro_block_count:0, row_count:0, rowkey_column_cnt:5, column_cnt:10, column_checksums:[], data_checksum:0, occupy_size:0, original_size:0, max_merged_trans_version:1726241132764173999, ddl_scn:{val:0, v:0}, contain_uncommitted_row:false, is_meta_root:false, compressor_type:1, encrypt_id:0, master_key_id:0, recycle_version:0, nested_offset:0, nested_size:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}, table_key={tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_merge_info={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:2019, merge_start_time:1727337660727197, merge_finish_time:1727337660729216, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"cost_time", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"incremental_row_count", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:1, snapshot_version:0, start_scn:1726211867305630000, end_scn:1726241548806527000}, macro_id_list:"", comment:""}) [2024-09-26 16:01:00.729791] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=2] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.729797] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337660729796, task_id:YB42C0A8002D-0006230056D8C722-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.729800] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=3] schedule one task(task=0x7f9264d82910, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.729832] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.729835] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.729837] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:00.729840] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.729845] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.729850] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.729856] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=215, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:6}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a68060, count:3, allocator:0x7f92636447d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9264a68320, local_datums:0x7f9264a68320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:01:00.729883] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.729917] ERROR issue_dba_error (ob_log.cpp:1875) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.729920] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.729927] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=0] partition merge iter row count(i=0, row_count=9, ghost_row_count=0, pkey={tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337660634217, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.729945] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660729824, end_time=1727337660729945, runtime=121, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71E-0-0, dag_ret:0, dag_status:2, start_time:1727337660728917, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.729954] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1037, dag_cnt=12, dag_cnts_[dag.get_type()]=12, &dag=0x7f9264d81090, dag={ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660728917, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.729961] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71E-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660728912, task_id:YB42C0A8002D-0006230056D8C71E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.729979] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] succeed to add sys task(task={start_time:1727337660729978, task_id:YB42C0A8002D-0006230056D8C723-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.729981] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=2] schedule one task(task=0x7f9264d82e40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.729996] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.729998] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.730000] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:00.730004] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.730007] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.730012] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.736759] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337660347134]}) [2024-09-26 16:01:00.737723] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=12] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:01:00.738687] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:00.738697] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.740441] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337660634272, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.740480] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=38] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.740489] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337660634252, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.740499] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=14] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.740504] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.740503] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=14] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.740518] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.740508] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337660634308, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.740510] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101015}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.740522] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.740525] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.740528] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:154}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.740542] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=13] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.740553] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=10] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.740529] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101015}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101015}, table_count:1, [{i:0, table_key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660729662, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=10.85ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.740560] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101063}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.740574] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=36] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C721-0-0, dag_ret:0, dag_status:2, start_time:1727337660729487, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab490, output_block_cnt_arr:0x7f92684ab498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337660729662, estimated_finish_time:1727337677740574}) [2024-09-26 16:01:00.740582] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d823e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C721-0-0, dag_ret:0, dag_status:2, start_time:1727337660729487, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.740588] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=5] task finish process(ret=0, start_time=1727337660729660, end_time=1727337660740588, runtime=10928, *this={this:0x7f9264d823e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C721-0-0, dag_ret:0, dag_status:2, start_time:1727337660729487, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.740581] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:101063}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f92a23a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101063}, table_count:1, [{i:0, table_key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660729992, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=10.56ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.740613] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=2] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.740616] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C723-0-0, dag_ret:0, dag_status:2, start_time:1727337660729981, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a74a0, output_block_cnt_arr:0x7f92a23a74a8, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337660729992, estimated_finish_time:1727337677740616}) [2024-09-26 16:01:00.740625] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d82e40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C723-0-0, dag_ret:0, dag_status:2, start_time:1727337660729981, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.740582] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:154}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:154}, table_count:1, [{i:0, table_key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8623ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660729715, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=10.81ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.740630] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=5] task finish process(ret=0, start_time=1727337660729991, end_time=1727337660740630, runtime=10639, *this={this:0x7f9264d82e40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C723-0-0, dag_ret:0, dag_status:2, start_time:1727337660729981, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.740641] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=51] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C720-0-0, dag_ret:0, dag_status:2, start_time:1727337660729458, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad480, output_block_cnt_arr:0x7f92684ad488, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337660729715, estimated_finish_time:1727337677740641}) [2024-09-26 16:01:00.740649] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d81eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C720-0-0, dag_ret:0, dag_status:2, start_time:1727337660729458, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.740654] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=5] task finish process(ret=0, start_time=1727337660729713, end_time=1727337660740654, runtime=10941, *this={this:0x7f9264d81eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C720-0-0, dag_ret:0, dag_status:2, start_time:1727337660729458, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.740707] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101015, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c6e060, count:6, allocator:0x7f92864427d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9262c6e390, local_datums:0x7f9262c6e390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:01:00.740793] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.740799] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.740809] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337660634272, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.740830] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660740675, end_time=1727337660740830, runtime=155, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C721-0-0, dag_ret:0, dag_status:2, start_time:1727337660729487, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.740840] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=11352, dag_cnt=11, dag_cnts_[dag.get_type()]=11, &dag=0x7f9264d82020, dag={ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C721-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660729487, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.740848] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C721-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660729483, task_id:YB42C0A8002D-0006230056D8C721-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.740930] INFO [STORAGE] build_new_table_store (ob_tablet_table_store.cpp:1206) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=19] succeed to build new table store(major_tables_={cnt:1, is_inited:true}, minor_tables_={cnt:3, is_inited:true}, memtables_={count:1}, PRINT_TS(*this)={ObTabletTableStore_Pretty:this:0x7f92a9fcfca8, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, memtables:{count:1}, is_ready_for_read:true, table_array:{ [2879] [ YB42C0A8002D-0006230056D8C71D-0-0 ] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [2879] [ YB42C0A8002D-0006230056D8C71D-0-0 ] MAJOR MAJOR 1725120003420849000 1725120003420849000 {val:0, v:0} {val:1726077600415496000, v:0} 0 false [2879] [ YB42C0A8002D-0006230056D8C71D-0-0 ] MINOR MINOR 1726178585269925002 1726178585269925002 {val:1726077259159067000, v:0} {val:1726178585269925002, v:0} 0 false [2879] [ YB42C0A8002D-0006230056D8C71D-0-0 ] MINI 1726211867106157000 1726211867106157000 {val:1726178585269925002, v:0} {val:1726211867305630000, v:0} 0 false [2879] [ YB42C0A8002D-0006230056D8C71D-0-0 ] MINI 1726241132764173999 1726241132764173999 {val:1726211867305630000, v:0} {val:1726241548806527000, v:0} 0 false [2879] [ YB42C0A8002D-0006230056D8C71D-0-0 ] MEM ACTIVE 9223372036854775807 18446744073709547609 {val:1726241548806527000, v:0} {val:4611686018427387903, v:0} 3 unused }}) [2024-09-26 16:01:00.740961] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337660634291, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.740972] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.740984] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.740987] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.740987] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337660634235, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.740993] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101016}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.741000] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753355744, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753355744, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.741015] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.741019] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.741010] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:101016}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101016}, table_count:1, [{i:0, table_key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660729828, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=11.16ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.741025] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:456}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753355744, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.741035] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C722-0-0, dag_ret:0, dag_status:2, start_time:1727337660729800, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d490, output_block_cnt_arr:0x7f9264b1d498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337660729828, estimated_finish_time:1727337677741035}) [2024-09-26 16:01:00.741041] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d82910, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C722-0-0, dag_ret:0, dag_status:2, start_time:1727337660729800, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.741046] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=5] task finish process(ret=0, start_time=1727337660729826, end_time=1727337660741046, runtime=11220, *this={this:0x7f9264d82910, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C722-0-0, dag_ret:0, dag_status:2, start_time:1727337660729800, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.741044] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:456}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753355744, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:456}, table_count:1, [{i:0, table_key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8634c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660729507, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=11.52ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.741087] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=35] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71F-0-0, dag_ret:0, dag_status:2, start_time:1727337660729452, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5490, output_block_cnt_arr:0x7f9287db5498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337660729507, estimated_finish_time:1727337677741087}) [2024-09-26 16:01:00.741094] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d81980, type:15, status:2, dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71F-0-0, dag_ret:0, dag_status:2, start_time:1727337660729452, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.741100] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=5] task finish process(ret=0, start_time=1727337660729505, end_time=1727337660741099, runtime=11594, *this={this:0x7f9264d81980, type:15, status:2, dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71F-0-0, dag_ret:0, dag_status:2, start_time:1727337660729452, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.741128] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=3] schedule one task(task=0x7f9264b56030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.741136] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=5] schedule one task(task=0x7f9287c663c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.741139] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=2] schedule one task(task=0x7f9264b36030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.741144] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=2] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.741151] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660741149, task_id:YB42C0A8002D-0006230056D8C724-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.741156] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=4] schedule one task(task=0x7f9264d83370, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.741196] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=101016, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:6, allocator:0x7f926912c7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:01:00.741219] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=154, range=0x7f92ab152200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a64060, count:5, allocator:0x7f9269f787d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a64360, local_datums:0x7f9264a64360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:01:00.741281] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.741285] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.741284] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.741293] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337660634291, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.741311] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337660741168, end_time=1727337660741311, runtime=143, *this={this:0x7f9264b36030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C722-0-0, dag_ret:0, dag_status:2, start_time:1727337660729800, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.741321] ERROR issue_dba_error (ob_log.cpp:1875) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.741320] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=11519, dag_cnt=10, dag_cnts_[dag.get_type()]=10, &dag=0x7f9264d82550, dag={ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C722-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660729800, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.741327] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C722-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660729796, task_id:YB42C0A8002D-0006230056D8C722-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.741324] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.741334] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337660634252, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.741346] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337660741345, task_id:YB42C0A8002D-0006230056D8C725-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.741350] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=3] schedule one task(task=0x7f9264d83920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.741353] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660741184, end_time=1727337660741353, runtime=169, *this={this:0x7f9287c663c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C720-0-0, dag_ret:0, dag_status:2, start_time:1727337660729458, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.741375] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=11917, dag_cnt=9, dag_cnts_[dag.get_type()]=9, &dag=0x7f9264d81af0, dag={ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C720-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660729458, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.741383] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C720-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660729455, task_id:YB42C0A8002D-0006230056D8C720-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.741413] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660741409, task_id:YB42C0A8002D-0006230056D8C726-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.741420] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=6] schedule one task(task=0x7f9264d83e50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.741475] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101063, range=0x7f92a9b52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a5e060, count:8, allocator:0x7f9269f787d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264a5e3e0, local_datums:0x7f9264a5e3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:00.741500] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.741539] ERROR issue_dba_error (ob_log.cpp:1875) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.741542] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.741551] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337660634308, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.741571] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337660741425, end_time=1727337660741571, runtime=146, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C723-0-0, dag_ret:0, dag_status:2, start_time:1727337660729981, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.741579] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=11598, dag_cnt=8, dag_cnts_[dag.get_type()]=8, &dag=0x7f9264d82a80, dag={ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C723-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660729981, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.741582] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.741586] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C723-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337660729978, task_id:YB42C0A8002D-0006230056D8C723-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.741590] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.741595] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:00.741615] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.741617] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.741620] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.741623] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:00.741623] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=5] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.741629] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.741634] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.741633] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706539141067, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f926e107fe0, is_inited:true, wash_score:1727337660634312000, hold_ref_cnt:true, ref_cnt:2, version:2, length:516, tablet_addr:[809012-1445-0-2305843657754198016-2](offset=442368,size=604,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:330}, data_tablet_id:{id:330}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706539141067, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3803410633, row_count:15330}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f926e108704, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844604794474496-0](offset=430080,size=1486,type=2,seq=0), type:"MAJOR", tablet_id:{id:330}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562805980000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843948738221518-0](offset=431566,size=875,type=2,seq=0), type:"MINI", tablet_id:{id:330}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986144852000, upper_trans_version:1726149986144852000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337660634366, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdade0, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220705914}, this:0x7f92a3bdade0, timestamp:1727334220705914, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:79, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265269081598001, v:0}, rec_scn:{val:1726264907253071001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696413696-2](offset=434176,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843189602754892-0](offset=434508,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879780852-0](offset=434676,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446300886-0](offset=434902,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc9670, 0x7f92a3bdade0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:00.741686] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=52] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.741693] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.741707] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=456, range=0x7f92aad52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:7, allocator:0x7f9269f787d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f9263a383b0, local_datums:0x7f9263a383b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:01:00.741815] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=23][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.741882] ERROR issue_dba_error (ob_log.cpp:1875) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=11][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.741887] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.741901] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=0] partition merge iter row count(i=0, row_count=45, ghost_row_count=0, pkey={tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337660634235, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.741935] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660741661, end_time=1727337660741935, runtime=274, *this={this:0x7f9264b56030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71F-0-0, dag_ret:0, dag_status:2, start_time:1727337660729452, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.741954] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=12502, dag_cnt=7, dag_cnts_[dag.get_type()]=7, &dag=0x7f9264d815c0, dag={ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660729452, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.741980] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71F-0-0] [lt=23] succeed to del sys task(removed_task={start_time:1727337660729448, task_id:YB42C0A8002D-0006230056D8C71F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.741984] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.741989] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.741992] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:00.741999] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.742004] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.742013] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.742012] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337660742010, task_id:YB42C0A8002D-0006230056D8C727-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.742018] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=5] schedule one task(task=0x7f9264d84380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.742027] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337660742026, task_id:YB42C0A8002D-0006230056D8C728-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.742031] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=4] schedule one task(task=0x7f9264d848b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.742052] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.742054] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.742056] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.742058] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.742058] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:00.742061] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:00.742063] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.742065] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.742067] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.742075] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.742068] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.742084] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=13] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.742494] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2256502784, global_cache_size=14565376, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:00.742544] INFO [COMMON] wash (ob_kvcache_store.cpp:335) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash memory, (tenant_id=1, cache_size=14565376, lower_mem_limit=2147483648, upper_mem_limit=2147483648, min_wash_size=0, max_wash_size=0, mem_usage=1307783168, reserve_mem=137180449, wash_size=2256502784) [2024-09-26 16:01:00.742625] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash time detail, (compute_wash_size_time=68, refresh_score_time=17, wash_time=9) [2024-09-26 16:01:00.743208] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=2] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=1100995, clean_num=31457) [2024-09-26 16:01:00.748858] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337660634403, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.748874] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=25, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.748878] INFO [MDS] init_for_evict_medium_info (ob_tablet_mds_data.cpp:414) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=16] succeeded to init mds data(ret=0, finish_medium_scn=1726077600415496000, src_medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, src_extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}) [2024-09-26 16:01:00.748890] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.748894] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.748902] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:332}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:27, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.748924] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:332}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:332}, table_count:1, [{i:0, table_key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8648b00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660742046, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=6.85ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.748952] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C728-0-0, dag_ret:0, dag_status:2, start_time:1727337660742031, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5490, output_block_cnt_arr:0x7f9287db5498, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337660742046, estimated_finish_time:1727337677748952}) [2024-09-26 16:01:00.748959] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d848b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C728-0-0, dag_ret:0, dag_status:2, start_time:1727337660742031, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.748965] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=6] task finish process(ret=0, start_time=1727337660742045, end_time=1727337660748965, runtime=6920, *this={this:0x7f9264d848b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C728-0-0, dag_ret:0, dag_status:2, start_time:1727337660742031, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.748986] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=3] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.749007] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337660634328, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.749018] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=15, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.749029] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.748917] INFO [STORAGE] init_for_merge (ob_tablet.cpp:363) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=11] succeeded to init tablet for mini/minor/major merge(ret=0, param={sstable:0x7f92728f0128, snapshot_version:1726241132764173999, clog_checkpoint_scn:{val:1726241548806527000, v:0}, multi_version_start:1726178585269925002, need_report:false, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, rebuild_seq:0, update_with_major_flag:false, need_check_sstable:false, ddl_info:{keep_old_ddl_sstable:true, ddl_start_scn:{val:0, v:0}, ddl_commit_scn:{val:0, v:0}, ddl_checkpoint_scn:{val:0, v:0}, ddl_snapshot_version:0, ddl_execution_id:-1, data_format_version:0}, allow_duplicate_sstable:false, merge_type:"MINI_MERGE", need_check_transfer_seq:true, transfer_seq:0}, old_tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337660633234000, hold_ref_cnt:true, ref_cnt:3, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337660634194, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}, this={this:0x7f92728f2060, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x7f92a9fd0450, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726241548806527000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726241132764173999, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92728f2670, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}, {i:2, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726241132764173999, upper_trans_version:1726241132764173999}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x7f92728f2520, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f92728f3880, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f92728f3950, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{redefined:false, snapshot_version:0, schema_version:0, data_tablet_id:{id:0}, hidden_tablet_id:{id:0}, lob_meta_tablet_id:{id:50121}, lob_piece_tablet_id:{id:60121}, is_old_mds:true}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:01:00.749032] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.749036] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:452}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:17, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.749052] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:452}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9263a4b330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:452}, table_count:1, [{i:0, table_key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8652d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660741574, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=7.46ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.749049] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=115] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9263a42060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}}, ref_cnt:0, upper_trans_version:1725120003420849000, timestamp:0}, this:0x7f9263a42060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1725120003420849000, max_merged_trans_version:1725120003420849000, data_macro_block_count:1, nested_size:4096, nested_offset:589824, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:84, row_count_:14, occupy_size_:1604, original_size_:1914, data_checksum_:3587602083, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:1, data_micro_block_count_:1, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:2, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1725120003420849000, max_merged_trans_version_:1725120003420849000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9263a42310, column_checksum_count:10, data_root_info:{addr:[0-0-0-3458764933653594112-0](offset=0,size=391,type=3,seq=0), block_data:{buf:0x7f9263a42360, size:391, extra_buf:0x7f9263a424e7, extra_size:424, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-3458765009889263616-0](offset=0,size=462,type=3,seq=0), block_data:{buf:0x7f9263a4268f, size:462, extra_buf:null, extra_size:0, type:1}}, data_block_count:1, other_block_count:0, linked_block_count:0, data_block_ids:0x7f9263a4285d, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:true, nested_offset:589824, nested_size:4096}}, valid_for_reading:true}) [2024-09-26 16:01:00.749070] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C724-0-0, dag_ret:0, dag_status:2, start_time:1727337660741155, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a4b480, output_block_cnt_arr:0x7f9263a4b488, concurrent_cnt:1, estimate_row_cnt:10, estimate_occupy_size:0, latest_update_ts:1727337660741574, estimated_finish_time:1727337677749070}) [2024-09-26 16:01:00.749076] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d83370, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C724-0-0, dag_ret:0, dag_status:2, start_time:1727337660741155, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.749073] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=21] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9263a42060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}}, ref_cnt:0, upper_trans_version:1726178585269925002, timestamp:0}, this:0x7f9263a42060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726178585269925002, max_merged_trans_version:1726178585269925002, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726178585269925002, v:0}, meta:{basic_meta:{version:1, length:94, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726178585269925002, max_merged_trans_version_:1726178585269925002, recycle_version:1726077600415496000, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726178585269925002, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:1, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9263a42310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:01:00.749081] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=5] task finish process(ret=0, start_time=1727337660741571, end_time=1727337660749081, runtime=7510, *this={this:0x7f9264d83370, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C724-0-0, dag_ret:0, dag_status:2, start_time:1727337660741155, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.749084] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=10] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9263a42060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}}, ref_cnt:0, upper_trans_version:1726211867106157000, timestamp:0}, this:0x7f9263a42060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726211867106157000, max_merged_trans_version:1726211867106157000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726211867305630000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726211867106157000, max_merged_trans_version_:1726211867106157000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726211867305630000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9263a42310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:01:00.749095] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337660634349, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.749094] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=10] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9263a42060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9263a42060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9263a42310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:01:00.749104] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=8] update column cnt(column_cnt_=26, store_column_cnt_=26, column_cnt_=26, column_array_.count()=0) [2024-09-26 16:01:00.749110] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=5] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92a23a7480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=26, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.749119] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.749121] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.749126] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92a23a7480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:328}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:28, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.749141] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.749144] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf9de5b1 0xf9dcfd6 0xe51c6fb 0xe519c03 0xe518f89 0xe516f8c 0xf01158d 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.749140] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:328}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92a23a7480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:328}, table_count:1, [{i:0, table_key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e072bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660741977, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=7.15ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.749155] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C725-0-0, dag_ret:0, dag_status:2, start_time:1727337660741349, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92a23a75d0, output_block_cnt_arr:0x7f92a23a75d8, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337660741977, estimated_finish_time:1727337677749155}) [2024-09-26 16:01:00.749160] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d83920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C725-0-0, dag_ret:0, dag_status:2, start_time:1727337660741349, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.749163] INFO [STORAGE] reset (ob_tablet.cpp:159) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=0] reset tablet(this=0x7f92728f2060, ls_id={id:1}, tablet_id={id:121}, lbt()="0x121f1f9c 0xe52cc09 0xe52c6c5 0xe52bd13 0x4fd348f 0xf01110c 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:00.749165] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=5] task finish process(ret=0, start_time=1727337660741974, end_time=1727337660749165, runtime=7191, *this={this:0x7f9264d83920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C725-0-0, dag_ret:0, dag_status:2, start_time:1727337660741349, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.749176] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=4] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.749182] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=3] schedule one task(task=0x7f9287c663c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.749171] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1250) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=0][errcode=-4184] sstable merge finish(ret=-4184, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71D-0-0, dag_ret:0, dag_status:2, start_time:1727337660727185, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1480, output_block_cnt_arr:0x7f92728f1488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337660727197, estimated_finish_time:1727337677728665}, compaction_filter:NULL, time_guard:total=2.55ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, task={this:0x7f9264d85170, type:16, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71D-0-0, dag_ret:0, dag_status:2, start_time:1727337660727185, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:00.749218] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337660729691, end_time=1727337660749218, runtime=19527, *this={this:0x7f9264d85170, type:16, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71D-0-0, dag_ret:0, dag_status:2, start_time:1727337660727185, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:00.749226] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=22041, dag_cnt=6, dag_cnts_[dag.get_type()]=6, &dag=0x7f9264d80b60, dag={ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C71D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660727185, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}) [2024-09-26 16:01:00.749234] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C71D-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660727181, task_id:YB42C0A8002D-0006230056D8C71D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.749264] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337660749261, task_id:YB42C0A8002D-0006230056D8C729-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.749269] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=4] schedule one task(task=0x7f9264d84de0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.749313] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=328, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:26, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:28}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685e6060, count:5, allocator:0x7f92864407d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:28, datum_buffer:{capacity:32, datums:0x7f92685e6360, local_datums:0x7f92685e6360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,]}) [2024-09-26 16:01:00.749319] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=452, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:15, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:17}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:5, allocator:0x7f92636167d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:17, datum_buffer:{capacity:32, datums:0x7f9263a3e360, local_datums:0x7f9263a3e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,]}) [2024-09-26 16:01:00.749325] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337660634383, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.749344] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=18] update column cnt(column_cnt_=29, store_column_cnt_=29, column_cnt_=29, column_array_.count()=0) [2024-09-26 16:01:00.749349] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=4] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9287db7490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.749363] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.749367] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.749367] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=22][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.749373] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db7490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:329}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.749394] ERROR issue_dba_error (ob_log.cpp:1875) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.749401] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.749398] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.749393] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:329}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9287db7490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:329}, table_count:1, [{i:0, table_key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e089f20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660742051, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=7.32ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.749405] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.749408] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=1] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337660634349, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.749414] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=0] partition merge iter row count(i=0, row_count=10, ghost_row_count=0, pkey={tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337660634328, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.749420] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C727-0-0, dag_ret:0, dag_status:2, start_time:1727337660742018, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db75f0, output_block_cnt_arr:0x7f9287db75f8, concurrent_cnt:1, estimate_row_cnt:62, estimate_occupy_size:0, latest_update_ts:1727337660742051, estimated_finish_time:1727337677749419}) [2024-09-26 16:01:00.749426] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d84380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C727-0-0, dag_ret:0, dag_status:2, start_time:1727337660742018, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.749429] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660749283, end_time=1727337660749429, runtime=146, *this={this:0x7f9287c663c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C725-0-0, dag_ret:0, dag_status:2, start_time:1727337660741349, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.749432] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=5] task finish process(ret=0, start_time=1727337660742049, end_time=1727337660749432, runtime=7383, *this={this:0x7f9264d84380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C727-0-0, dag_ret:0, dag_status:2, start_time:1727337660742018, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.749434] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660749286, end_time=1727337660749434, runtime=148, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C724-0-0, dag_ret:0, dag_status:2, start_time:1727337660741155, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.749439] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=8090, dag_cnt=5, dag_cnts_[dag.get_type()]=5, &dag=0x7f9264d83560, dag={ObIDag:{this:0x7f9264d83560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C725-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660741349, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.749465] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C725-0-0] [lt=8] succeed to del sys task(removed_task={start_time:1727337660741345, task_id:YB42C0A8002D-0006230056D8C725-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.749479] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.749483] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:00.749495] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=5][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:00.749502] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:00.749506] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:00.749515] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.749560] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337660634366, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.749571] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=9] update column cnt(column_cnt_=12, store_column_cnt_=12, column_cnt_=12, column_array_.count()=0) [2024-09-26 16:01:00.749574] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9263a45490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.749594] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.749597] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.749603] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:330}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.749622] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:330}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9263a45490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:330}, table_count:1, [{i:0, table_key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e107fe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660741612, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=7.99ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.749652] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C726-0-0, dag_ret:0, dag_status:2, start_time:1727337660741419, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a455f0, output_block_cnt_arr:0x7f9263a455f8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337660741612, estimated_finish_time:1727337677749652}) [2024-09-26 16:01:00.749659] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d83e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C726-0-0, dag_ret:0, dag_status:2, start_time:1727337660741419, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.749664] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=5] task finish process(ret=0, start_time=1727337660741610, end_time=1727337660749664, runtime=8054, *this={this:0x7f9264d83e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C726-0-0, dag_ret:0, dag_status:2, start_time:1727337660741419, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.749682] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=8527, dag_cnt=4, dag_cnts_[dag.get_type()]=4, &dag=0x7f9264d82fb0, dag={ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C724-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660741155, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.749690] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C724-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660741149, task_id:YB42C0A8002D-0006230056D8C724-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.749748] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=332, range=0x7f92a9952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:25, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:27}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b0060, count:6, allocator:0x7f92c183a7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:27, datum_buffer:{capacity:32, datums:0x7f92684b0390, local_datums:0x7f92684b0390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,]}) [2024-09-26 16:01:00.749810] ERROR issue_dba_error (ob_log.cpp:1875) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.749813] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.749822] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337660634403, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.749838] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660749698, end_time=1727337660749838, runtime=140, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C728-0-0, dag_ret:0, dag_status:2, start_time:1727337660742031, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.749847] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=7816, dag_cnt=3, dag_cnts_[dag.get_type()]=3, &dag=0x7f9264d844f0, dag={ObIDag:{this:0x7f9264d844f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C728-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660742031, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.749854] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C728-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660742026, task_id:YB42C0A8002D-0006230056D8C728-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.749884] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=2] schedule one task(task=0x7f9272926030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.749936] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=330, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:7, allocator:0x7f926912a7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9263b5e3b0, local_datums:0x7f9263b5e3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:01:00.750120] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.750126] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.750137] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=0] partition merge iter row count(i=0, row_count=141, ghost_row_count=0, pkey={tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337660634366, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.750158] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660749906, end_time=1727337660750158, runtime=252, *this={this:0x7f9272926030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C726-0-0, dag_ret:0, dag_status:2, start_time:1727337660741419, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.750181] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=8761, dag_cnt=2, dag_cnts_[dag.get_type()]=2, &dag=0x7f9264d83a90, dag={ObIDag:{this:0x7f9264d83a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C726-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660741419, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.750188] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C726-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660741409, task_id:YB42C0A8002D-0006230056D8C726-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.750200] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=4] schedule one task(task=0x7f9264b56030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.750251] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=329, range=0x7f92ab0d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:6, allocator:0x7f92a339c7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9263a38390, local_datums:0x7f9263a38390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:01:00.750371] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.750416] ERROR issue_dba_error (ob_log.cpp:1875) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.750421] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.750440] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=0] partition merge iter row count(i=0, row_count=47, ghost_row_count=0, pkey={tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337660634383, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.750458] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660750213, end_time=1727337660750458, runtime=245, *this={this:0x7f9264b56030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C727-0-0, dag_ret:0, dag_status:2, start_time:1727337660742018, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.750466] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=1] dag finished(dag_ret=-4184, runtime=8448, dag_cnt=1, dag_cnts_[dag.get_type()]=1, &dag=0x7f9264d83fc0, dag={ObIDag:{this:0x7f9264d83fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C727-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660742018, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.750473] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C727-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660742010, task_id:YB42C0A8002D-0006230056D8C727-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.753605] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337660634421, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:00.753620] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92728f7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:00.753642] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:00.753645] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:00.753653] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101067}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:00.753674] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:101067}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92728f7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101067}, table_count:1, [{i:0, table_key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b86a2d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337660749473, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.18ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:00.753707] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C729-0-0, dag_ret:0, dag_status:2, start_time:1727337660749268, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f74a0, output_block_cnt_arr:0x7f92728f74a8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337660749473, estimated_finish_time:1727337677753706}) [2024-09-26 16:01:00.753715] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d84de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C729-0-0, dag_ret:0, dag_status:2, start_time:1727337660749268, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.753721] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=5] task finish process(ret=0, start_time=1727337660749459, end_time=1727337660753720, runtime=4261, *this={this:0x7f9264d84de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C729-0-0, dag_ret:0, dag_status:2, start_time:1727337660749268, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.753730] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=4] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:00.753878] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101067, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d2e060, count:8, allocator:0x7f928643e7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d2e3e0, local_datums:0x7f9287d2e3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:00.754074] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=18][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:00.754114] ERROR issue_dba_error (ob_log.cpp:1875) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:00.754118] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:00.754127] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=0] partition merge iter row count(i=0, row_count=227, ghost_row_count=0, pkey={tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337660634421, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:00.754150] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337660753844, end_time=1727337660754150, runtime=306, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C729-0-0, dag_ret:0, dag_status:2, start_time:1727337660749268, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:00.754161] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4893, dag_cnt=0, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264d84a20, dag={ObIDag:{this:0x7f9264d84a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C729-0-0, dag_ret:-4184, dag_status:5, start_time:1727337660749268, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:00.754169] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C729-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337660749261, task_id:YB42C0A8002D-0006230056D8C729-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:00.768798] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:00.768811] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.775267] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:00.792454] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=14] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:01:00.798284] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:00.798323] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337660798259) [2024-09-26 16:01:00.798331] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337660697670, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:00.798354] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223390034756, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.798361] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:00.798370] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:00.798377] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:00.798380] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:00.798387] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.798390] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.798392] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337660798335) [2024-09-26 16:01:00.800184] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14] success to notify_need_writing_throttling True [2024-09-26 16:01:00.800195] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.821710] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.821750] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.821753] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.821760] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.821765] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.821768] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.821770] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.821775] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.821775] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.821780] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.821785] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:00.830482] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=33] success to notify_need_writing_throttling True [2024-09-26 16:01:00.830499] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.837346] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337660347134]}) [2024-09-26 16:01:00.848171] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:00.848182] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337660848169], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:00.848189] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337660848169], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:00.848193] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:00.848197] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:00.860748] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [2925][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=11] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f92b77d02b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-09-26 16:01:00.860753] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:01:00.860760] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.875454] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=6][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337661804787, now=[mts=1727337659875006]) [2024-09-26 16:01:00.875467] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=12][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337661804787, now=[mts=1727337659875006], now0=[mts=1727337659875006], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:00.875487] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=18][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a312b360, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500836, assoc_session_id:3221500836, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337659874329, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:00.875506] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=18][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22fdff8, id:3221500836, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312b360}) [2024-09-26 16:01:00.875515] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500836, *tx_desc={this:0x7f92a312b360, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500836, assoc_session_id:3221500836, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337659874329, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:00.875522] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500836, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a312b360, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500836, assoc_session_id:3221500836, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337659874329, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337659874983, use_das=false, nested_level=0, session={this:0x7f92a22fdff8, id:3221500836, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312b360}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:00.875542] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=19][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:00.875547] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:00.875549] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:00.875552] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:00.875555] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:00.875562] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=5][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:00.875567] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=999220, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337661874787) [2024-09-26 16:01:00.875572] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:00.875580] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:00.875582] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:00.875583] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:00.875589] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:00.875629] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:01:00.875632] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:00.891066] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=4] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-09-26 16:01:00.891080] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=9] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=692) [2024-09-26 16:01:00.891099] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:00.891104] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.898842] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:00.898886] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337660898837) [2024-09-26 16:01:00.898893] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337660798335, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:00.898913] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223390135353, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.898921] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:00.898931] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:00.898937] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:00.898940] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:00.898948] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.898950] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.898953] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337660898897) [2024-09-26 16:01:00.904651] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337660904650, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006868, add_bytes: 0 [2024-09-26 16:01:00.921332] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:00.921343] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.938877] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337660848169]}) [2024-09-26 16:01:00.943927] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:00.944068] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash time detail, (compute_wash_size_time=73, refresh_score_time=36, wash_time=2) [2024-09-26 16:01:00.944709] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Cache wash clean map node details(ret=0, clean_node_count=4, clean_time=0, clean_start_pos=1132452, clean_num=31457) [2024-09-26 16:01:00.951612] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:00.951628] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.981875] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:00.981891] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:00.987747] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:00.992601] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:01:00.999487] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:00.999499] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337660999483) [2024-09-26 16:01:00.999505] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337660898897, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:00.999534] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223390234030, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:00.999542] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:00.999551] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:00.999557] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:00.999559] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:00.999566] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.999568] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:00.999570] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337660999510) [2024-09-26 16:01:01.011958] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14] success to notify_need_writing_throttling True [2024-09-26 16:01:01.011969] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.024991] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185932-0-0] [lt=38][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:01.025000] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185932-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:01.025009] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185932-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:01.025014] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185932-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:01.036713] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500841, support_ssl=false) [2024-09-26 16:01:01.036744] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221500841, sess.client_addr_="127.0.0.1:36382") [2024-09-26 16:01:01.036766] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500841, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:01.036798] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:01.036807] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2688][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:01.036845] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA5-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:01.037036] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA5-0-0] [lt=9][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:01.037045] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA5-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"o%dK))2_iqz54GGoz6>s"}, ret=-4043) [2024-09-26 16:01:01.037052] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA5-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:01.037056] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA5-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:01.037067] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA5-0-0] [lt=4] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:01.037090] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA5-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500841, proxy_sessid:0}) [2024-09-26 16:01:01.037095] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA5-0-0] [lt=4] mark session id unused(sessid=3221500841) [2024-09-26 16:01:01.037098] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA5-0-0] [lt=2][errcode=0] server close connection(sessid=3221500841, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:01.037101] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA5-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500841, proxy_sessid=0) [2024-09-26 16:01:01.037104] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA5-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:01.037106] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA5-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500841, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:01.037138] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500841, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337661036660, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:01.037148] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] kill and revert session(conn.sessid_=3221500841, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:01.037152] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=4] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500841, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337661036660, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:01.037156] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2688][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500841, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:01.039500] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337660848169]}) [2024-09-26 16:01:01.042421] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:01.042428] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.062311] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:01.062325] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:01:01.062329] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:01:01.062333] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:01:01.062336] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:01.062339] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:01.062342] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:01:01.062344] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:01.062346] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:01.062348] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:01.062350] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:01.062352] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:01.062355] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:01.062357] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:01.062363] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:01.062365] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:01.062368] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:01.062370] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:01.062372] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:01:01.062374] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:01.062377] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:01.062382] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:01.062388] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:01.062391] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:01.062392] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:01.062409] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:01.062415] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85896-0-0] [lt=4][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:01.062418] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:01:01.062421] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:01.062424] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85896-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:01.062426] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:01.062428] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337661062242, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:01.062432] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85896-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:01.062435] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85896-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:01:01.062459] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85896-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:01:01.062464] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85896-0-0] [lt=5][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:01:01.069770] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=5] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:01.070046] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:01.070053] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=5][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:01.070056] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:01.070059] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:01.070062] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:01.070064] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:01.070067] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:01.070069] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:01.070071] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:01.070072] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:01.070074] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:01.070076] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:01.070078] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:01.070080] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:01.070085] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:01.070087] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:01.070089] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:01.070091] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:01.070093] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:01:01.070095] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:01.070097] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:01.070101] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:01.070105] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:01.070107] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:01.070108] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:01.070113] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:01.070116] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:01.070118] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D84-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:01:01.070121] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:01.070123] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:01.070125] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:01.070127] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337661070001, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:01.070130] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:01.070171] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:01:01.073735] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=31] success to notify_need_writing_throttling True [2024-09-26 16:01:01.073742] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.089303] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=11] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:01.100065] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:01.100072] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:01.100078] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337661100061) [2024-09-26 16:01:01.100083] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337660999510, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:01.100088] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:01.100096] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:01.100099] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:01.100378] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:01.100388] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:01.100391] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:01.100394] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:01.100397] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:01.100400] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:01.100403] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:01.100406] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:01.100408] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:01.100410] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:01.100411] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:01.100414] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:01.100416] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:01.100419] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:01.100424] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:01.100426] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:01.100430] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:01.100432] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:01.100434] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:01.100437] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:01.100442] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=5][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:01.100448] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:01.100470] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:01.100472] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:01.100474] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:01.100479] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:01.100483] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:01.100485] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885778-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:01.100490] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:01.100493] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:01.100495] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:01.100497] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337661100326, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:01.100503] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:01.100505] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:01.100529] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:01.100533] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337661100532, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=440, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:01.100539] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:01.100543] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:01.100558] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223390336543, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:01.100563] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:01.100569] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:01.100575] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:01.100578] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:01.100583] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.100585] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.100588] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337661100546) [2024-09-26 16:01:01.104004] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:01.104009] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.134327] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:01:01.134334] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.134343] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:01.140073] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337660848169]}) [2024-09-26 16:01:01.146098] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:01.146198] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=50, refresh_score_time=15, wash_time=1) [2024-09-26 16:01:01.146798] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1163909, clean_num=31457) [2024-09-26 16:01:01.153776] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: SHARED_BLOCK_ReaderWriter_IO, mode: read, size: 4096.00, iops: 52.68, rt: 11664.34) [2024-09-26 16:01:01.153787] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:01.153806] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 12173.00) [2024-09-26 16:01:01.153809] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:01.153817] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:01.153820] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:01.164544] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:01.164549] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.192850] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:01:01.194831] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:01.194839] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.200650] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:01.200661] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337661200647) [2024-09-26 16:01:01.200665] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337661100085, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:01.200681] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223390436179, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:01.200686] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:01.200694] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:01.200698] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:01.200701] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:01.200708] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.200711] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.200712] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337661200669) [2024-09-26 16:01:01.210680] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:01:01.210692] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:01.210696] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:01:01.225089] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:01.225096] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.240690] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=25] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337660848169]}) [2024-09-26 16:01:01.240705] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D996-0-0] [lt=18][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337661240685}) [2024-09-26 16:01:01.256489] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:01:01.256497] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.257648] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:01.257660] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:01.257665] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:01.257668] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:01.257672] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:01.257687] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=15][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:01.257691] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:01.257693] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:01.257696] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:01.257698] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:01.257700] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:01.257702] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:01.257704] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:01.257706] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:01.257713] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:01.257716] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:01.257721] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=5][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:01.257724] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:01.257726] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:01:01.257729] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:01.257731] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:01.257737] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:01.257742] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:01.257745] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:01.257746] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:01.257752] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:01.257756] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:01.257759] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:01:01.257761] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:01.257764] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:01.257767] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:01.257769] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337661257583, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:01.257773] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:01.257776] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:01.257804] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:01.257808] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=4][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:01:01.257812] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=4][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:01.257814] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:01.257817] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:01.257821] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=3][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:01.257823] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A5-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:01.275880] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337660274804]) [2024-09-26 16:01:01.275892] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337660274804], now0=[mts=1727337660274804], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:01.275899] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337660274735, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:01.275915] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:01:01.275923] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337660274735, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:01.275930] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337660274735, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:01.275947] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:01.275951] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:01.275953] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:01.275956] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:01.275958] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:01:01.275962] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=23, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:01.275966] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=23000, remain_us=5718366, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=23, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:01:01.286593] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:01.286602] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.299312] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:23, local_retry_times:23, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:01.299338] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=21][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:01.299342] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:01.299343] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:01.299349] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:01.299386] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:01:01.299389] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:01.300811] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:01.300829] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337661300808) [2024-09-26 16:01:01.300836] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337661200668, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:01.300855] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223390536816, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:01.300860] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:01.300867] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:01.300873] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:01.300875] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:01.300882] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.300885] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.300887] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337661300839) [2024-09-26 16:01:01.316968] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:01.316977] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.319911] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337662249048, now=[mts=1727337660319307]) [2024-09-26 16:01:01.319921] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337662249048, now=[mts=1727337660319307], now0=[mts=1727337660319307], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:01.319928] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269014d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337660318707, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:01.319943] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=14][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269014d80}) [2024-09-26 16:01:01.319950] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269014d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337660318707, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:01.319957] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269014d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337660318707, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337660319277, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269014d80}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:01.319973] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:01.319978] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:01.319980] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:01.319983] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:01.319985] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:01.319988] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:01.319992] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=999056, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337662319048) [2024-09-26 16:01:01.319995] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:01.320001] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:01.320003] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:01.320004] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:01.320009] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:01.320041] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:01:01.320044] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:01.341271] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337660848169]}) [2024-09-26 16:01:01.348137] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:01.348152] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.348263] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:01.348367] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=54, refresh_score_time=15, wash_time=1) [2024-09-26 16:01:01.348560] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.348570] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337661348557], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:01.348577] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337661348557], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:01.348582] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=5082, get_gts_with_stc_cnt=11832, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-09-26 16:01:01.348971] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1195366, clean_num=31457) [2024-09-26 16:01:01.378467] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=33] success to notify_need_writing_throttling True [2024-09-26 16:01:01.378480] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.391121] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=9] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:01.392033] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=8] current monitor number(seq_=-1) [2024-09-26 16:01:01.393062] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:01:01.400877] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:01.400889] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337661400874) [2024-09-26 16:01:01.400895] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337661300839, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:01.400914] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223390635447, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:01.400920] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:01.400928] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:01.400934] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:01.400937] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:01.400945] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.400947] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.400949] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337661400898) [2024-09-26 16:01:01.408741] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:01.408748] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.420704] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:01:01.420728] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:01.420736] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:01.420739] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:01.420743] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:01.420749] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=4] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:01:01.420751] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:01.420753] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:01.420757] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:01.420760] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:01.420763] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:01.420765] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:01.420767] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:01.420771] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:01.420773] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:01.420776] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:01.420783] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:01.420786] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:01.420790] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:01.420794] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:01.420796] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-09-26 16:01:01.420802] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=2][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:01.420806] WDIAG [SERVER] do_process (obmp_query.cpp:771) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=4][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-09-26 16:01:01.420813] WDIAG [SERVER] do_process (obmp_query.cpp:903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=4][errcode=-5019] query failed(ret=-5019, session={this:0x7f92a230bdf8, id:3221487638, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-09-26 16:01:01.420824] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F54-0-0] [lt=7] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:01.426659] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D4272-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:01.438958] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:01:01.438966] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.441938] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337661348557]}) [2024-09-26 16:01:01.465378] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=17] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.465394] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.465400] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.465406] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=14] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.465410] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.465412] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.465416] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.465419] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.465421] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.465437] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337661465436, bytes: 14709442, bw: 0.003619 MB/s, add_ts: 1007043, add_bytes: 3822 [2024-09-26 16:01:01.466302] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.466307] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.466309] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.466323] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.466325] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.466327] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.466331] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.466333] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.466334] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.467288] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.467291] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.467293] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.467296] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.467298] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.467299] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.467302] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.467303] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.467308] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.467908] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.467913] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.467915] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.467919] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.467921] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.467922] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.467925] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.467926] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.467928] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.469238] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:01.469245] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.469268] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.469286] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.469290] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.469295] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.469298] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.469300] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.469303] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.469305] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.469306] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.470209] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.470214] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.470226] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:01.470230] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.470234] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.471198] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.471210] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.471233] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.473194] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.473200] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.473203] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.474143] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.474153] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.474171] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.475130] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.475141] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.475145] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.476087] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.476093] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.476096] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.476810] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.476820] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.476825] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.478052] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.478058] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.478062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.479017] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.479047] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.479053] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.479983] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.479989] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.479993] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.480979] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.480983] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.480986] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.481499] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.481504] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.481507] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.482920] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.482926] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.482930] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.483916] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.483923] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.483940] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.484469] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.484479] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.484483] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.485842] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.485859] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.485864] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.486855] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.486867] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.486870] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.487800] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.487806] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.487809] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.488789] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.488793] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.488796] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.489422] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.489430] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.489433] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.490127] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.490132] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.490134] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.490704] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.490716] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.490720] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.491701] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.491707] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.491710] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.491746] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=19] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:01:01.492685] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.492690] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.492694] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.493684] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.493701] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.493705] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.494659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.494667] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.494671] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.495618] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.495628] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.495632] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.496596] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.496613] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.496627] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.497017] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] ====== tenant freeze timer task ====== [2024-09-26 16:01:01.497026] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:01:01.497058] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337660633362, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337660633318, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337660633383, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:01:01.497083] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:01:01.497135] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E3-0-0] [lt=4] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:01:01.497140] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E3-0-0] [lt=4] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:01:01.497159] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E3-0-0] [lt=5] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:01:01.497164] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E3-0-0] [lt=4] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:01:01.497168] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E3-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:01:01.497170] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E3-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:01:01.497175] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E3-0-0] [lt=5][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:01:01.497184] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E3-0-0] [lt=8] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:01:01.497186] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E3-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:01.497191] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E3-0-0] [lt=5] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:01.497195] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E3-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:01.497226] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851E3-0-0] [lt=4] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:01:01.497237] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:01.499476] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:01.499484] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.501490] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:01.501499] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337661501488) [2024-09-26 16:01:01.501504] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337661400898, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:01.501520] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223390736068, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:01.501526] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:01.501533] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:01.501538] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:01.501541] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:01.501545] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.501548] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.501550] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337661501507) [2024-09-26 16:01:01.523094] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185933-0-0] [lt=47][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:01.523102] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185933-0-0] [lt=7][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:01.523110] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185933-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:01.523115] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185933-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:01.529802] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:01.529810] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.543484] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337661348557]}) [2024-09-26 16:01:01.547390] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6625, mem_used=12478464) [2024-09-26 16:01:01.549452] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:01.549569] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=51, refresh_score_time=28, wash_time=2) [2024-09-26 16:01:01.550157] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1226823, clean_num=31457) [2024-09-26 16:01:01.560034] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:01:01.560041] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.565927] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=9] gc stale ls task succ [2024-09-26 16:01:01.590348] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:01:01.590356] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.593292] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:01:01.602080] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.602079] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.602089] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.602093] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.602118] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:01.602126] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337661602116) [2024-09-26 16:01:01.602131] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337661501507, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:01.602137] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:01.602147] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:01.602150] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:01.602474] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:01.602486] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:01.602490] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:01.602493] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:01.602498] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=4][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:01.602501] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:01.602504] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:01.602506] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:01.602509] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:01.602511] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:01.602512] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:01.602514] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:01.602517] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:01.602519] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:01.602525] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:01.602529] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:01.602532] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:01.602535] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:01.602537] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:01.602540] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:01.602542] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:01.602555] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885779-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:01.602587] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337661602586, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=443, wlock_time=9, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:01.602597] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:01.602614] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223390836597, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:01.602619] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:01.602627] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:01.602632] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:01.602634] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:01.602659] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.602663] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.602665] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337661602602) [2024-09-26 16:01:01.620502] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:01:01.620513] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.643608] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337661348557]}) [2024-09-26 16:01:01.650900] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:01.650908] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.664599] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:01.681229] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:01.681239] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.695389] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:01.702675] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:01.702687] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337661702671) [2024-09-26 16:01:01.702692] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337661602134, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:01.702710] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223390938324, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:01.702716] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:01.702724] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:01.702729] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:01.702732] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:01.702738] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.702741] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.702743] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337661702696) [2024-09-26 16:01:01.711449] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29] success to notify_need_writing_throttling True [2024-09-26 16:01:01.711470] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.741746] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:01.741755] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.743701] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337661348557]}) [2024-09-26 16:01:01.743759] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=5][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4184, dropped:505, tid:2877}]) [2024-09-26 16:01:01.750310] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:01.750413] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=54, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:01.751046] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=3, clean_time=0, clean_start_pos=1258280, clean_num=31457) [2024-09-26 16:01:01.772065] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:01.772071] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.779856] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:01.793501] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:01:01.802337] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:01.802359] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.803288] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:01.803305] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337661803285) [2024-09-26 16:01:01.803310] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337661702696, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:01.803330] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223391038103, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:01.803338] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:01.803346] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:01.803351] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:01.803355] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:01.803362] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.803364] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.803366] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337661803313) [2024-09-26 16:01:01.821838] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.821847] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.821874] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.821878] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.821882] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.821890] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.821897] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.821969] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [2915][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=13] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-09-26 16:01:01.822005] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=30] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.822013] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.822020] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.822027] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:01.832583] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:01.832591] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.844344] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337661348557]}) [2024-09-26 16:01:01.849297] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:01.849306] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337661849295], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:01.849329] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337661849295], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:01.862893] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:01.862902] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.875610] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337661804787, now=[mts=1727337660875628]) [2024-09-26 16:01:01.875618] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337661804787, now=[mts=1727337660875628], now0=[mts=1727337660875628], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:01.875624] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a312b360, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500836, assoc_session_id:3221500836, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337660875548, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:01.875641] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22fdff8, id:3221500836, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312b360}) [2024-09-26 16:01:01.875649] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500836, *tx_desc={this:0x7f92a312b360, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500836, assoc_session_id:3221500836, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337660875548, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:01.875657] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500836, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a312b360, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500836, assoc_session_id:3221500836, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337660875548, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337659874983, use_das=false, nested_level=0, session={this:0x7f92a22fdff8, id:3221500836, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312b360}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:01.875674] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:01.875679] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:01.875682] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:01.875684] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:01.875687] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:01.875701] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=12][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:01.875706] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:01:01.875713] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:01.875715] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:01.875717] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:01:01.875723] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:01:01.875727] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=3] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000743) [2024-09-26 16:01:01.875729] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508B-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:01:01.875733] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:01.875737] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:01:01.875742] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:01:01.875745] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8620, start=1727337659874980, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:01.875748] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:01:01.875751] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:01.875796] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:01:01.875800] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:01:01.875803] WDIAG [SHARE] get_snapshot_gc_scn (ob_global_stat_proxy.cpp:164) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:01:01.875805] WDIAG [STORAGE] get_global_info (ob_tenant_freeze_info_mgr.cpp:811) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to get global info(ret=-4012, tenant_id=1) [2024-09-26 16:01:01.875808] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:954) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] failed to get global info(ret=-4012) [2024-09-26 16:01:01.875810] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1008) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to try update info(tmp_ret=-4012, tmp_ret="OB_TIMEOUT") [2024-09-26 16:01:01.875816] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1044) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3] success to update min reserved snapshot(reserved_snapshot=0, duration=1800, snapshot_gc_ts_=0) [2024-09-26 16:01:01.875819] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1071) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3] update reserved snapshot finished(cost_ts=4, reserved_snapshot=0) [2024-09-26 16:01:01.893121] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:01.893129] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.903373] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:01.903383] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337661903369) [2024-09-26 16:01:01.903389] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337661803313, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:01.903405] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223391139361, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:01.903411] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:01.903417] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:01.903422] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:01.903426] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:01.903432] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.903434] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:01.903436] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337661903391) [2024-09-26 16:01:01.912174] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337661912174, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007524, add_bytes: 0 [2024-09-26 16:01:01.923460] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:01:01.923466] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.944906] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337661849295]}) [2024-09-26 16:01:01.951955] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:01.952056] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=80, refresh_score_time=15, wash_time=1) [2024-09-26 16:01:01.952692] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1289737, clean_num=31457) [2024-09-26 16:01:01.953703] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:01.953709] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.984067] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:01.984078] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:01.988506] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] tx gc loop thread is running(MTL_ID()=1) [2024-09-26 16:01:01.988515] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] try gc retain ctx [2024-09-26 16:01:01.988907] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-09-26 16:01:01.988937] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:01.993736] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:01:02.003466] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:02.003477] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337662003463) [2024-09-26 16:01:02.003483] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337661903391, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:02.003500] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223391239435, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:02.003508] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:02.003516] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:02.003521] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:02.003524] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:02.003529] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.003532] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.003534] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337662003485) [2024-09-26 16:01:02.014205] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:02.014213] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.026346] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA6-0-0] [lt=25] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:02.026356] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA6-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:02.026386] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA6-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:02.026394] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:398) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=1] start to print blacklist info [2024-09-26 16:01:02.026423] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3] ls blacklist refresh finish(cost_time=461) [2024-09-26 16:01:02.026609] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185934-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:02.026615] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185934-0-0] [lt=6][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:02.026621] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185934-0-0] [lt=6][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:02.026626] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185934-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:02.038110] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500845, support_ssl=false) [2024-09-26 16:01:02.038135] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221500845, sess.client_addr_="127.0.0.1:36384") [2024-09-26 16:01:02.038141] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500845, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:02.038168] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:02.038176] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:02.038215] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA6-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:02.038419] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA6-0-0] [lt=9][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:02.038429] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA6-0-0] [lt=9][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"D=E#}n/>,^x_Te}hr"PL"}, ret=-4043) [2024-09-26 16:01:02.038436] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA6-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:02.038439] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA6-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:02.038470] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA6-0-0] [lt=3] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:02.038492] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA6-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500845, proxy_sessid:0}) [2024-09-26 16:01:02.038496] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA6-0-0] [lt=4] mark session id unused(sessid=3221500845) [2024-09-26 16:01:02.038501] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA6-0-0] [lt=3][errcode=0] server close connection(sessid=3221500845, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:02.038504] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA6-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500845, proxy_sessid=0) [2024-09-26 16:01:02.038507] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA6-0-0] [lt=3][errcode=-4016] session is null [2024-09-26 16:01:02.038510] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA6-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500845, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:02.038544] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500845, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337662038111, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:02.038566] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221500845, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:02.038570] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500845, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337662038111, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:02.038574] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500845, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:02.044510] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:02.044518] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.045146] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337661849295]}) [2024-09-26 16:01:02.063381] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:02.063392] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:01:02.063395] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:01:02.063399] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:01:02.063403] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:02.063405] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:02.063408] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:01:02.063410] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:02.063413] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:02.063414] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:02.063416] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:02.063418] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:02.063421] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:02.063423] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:02.063428] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:02.063431] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:02.063433] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:02.063445] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85897-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:02.070255] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:02.070551] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D86-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:02.070561] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D86-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:02.070596] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D86-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:02.074804] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:02.074811] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.090424] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=34] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:02.103815] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:02.103824] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:02.103830] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337662103812) [2024-09-26 16:01:02.103835] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337662003485, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:02.103840] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:02.103848] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:02.103851] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:02.104134] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577A-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:02.104162] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577A-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:02.104189] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337662104188, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=343, wlock_time=8, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:02.104198] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:02.104214] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223391338617, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:02.104220] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:02.104226] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:02.104231] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:02.104234] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:02.104240] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.104243] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.104245] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337662104202) [2024-09-26 16:01:02.105063] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:02.105073] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.135368] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:02.135378] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.135396] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:02.145069] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:141, tid:2464}]) [2024-09-26 16:01:02.146100] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337661849295]}) [2024-09-26 16:01:02.153061] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:02.153180] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=49, refresh_score_time=28, wash_time=2) [2024-09-26 16:01:02.153794] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1321194, clean_num=31457) [2024-09-26 16:01:02.157465] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 11093.00) [2024-09-26 16:01:02.157474] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:02.157486] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:02.157490] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:02.165501] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:02.165508] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.193973] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:01:02.195943] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:02.195952] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.201790] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=2] ====== check clog disk timer task ====== [2024-09-26 16:01:02.201800] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=8] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:01:02.201808] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:01:02.204724] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:02.204736] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337662204722) [2024-09-26 16:01:02.204742] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337662103838, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:02.204758] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223391439420, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:02.204764] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:02.204772] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:02.204777] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:02.204780] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:02.204786] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.204789] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.204791] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337662204745) [2024-09-26 16:01:02.226204] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:02.226220] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.246329] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337661849295]}) [2024-09-26 16:01:02.246347] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D9A2-0-0] [lt=7][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337662246324}) [2024-09-26 16:01:02.256766] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:02.256779] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.258768] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:02.258783] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:02.258787] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:02.258791] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:02.258794] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:02.258796] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:02.258800] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:02.258802] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:02.258804] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:02.258805] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:02.258807] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:02.258810] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:02.258812] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:02.258814] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:02.258819] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:02.258821] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:02.258824] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:02.258826] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:02.258828] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:01:02.258831] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:02.258833] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:02.258838] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:02.258843] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:02.258849] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=6][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:02.258851] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:02.258857] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:02.258861] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:02.258864] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:01:02.258869] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=5][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:02.258872] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:02.258874] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:02.258877] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337662258706, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:02.258880] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:02.258883] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:02.258910] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=3][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:02.258915] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=4][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:01:02.258917] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:02.258920] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:02.258922] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=1][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:02.258925] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:02.258927] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A6-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:02.287726] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:02.287736] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.300418] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337661299385]) [2024-09-26 16:01:02.300429] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337661299385], now0=[mts=1727337661299385], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:02.300436] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337661299317, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:02.300468] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=31][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:01:02.300476] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337661299317, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:02.300483] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337661299317, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:02.300501] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:02.300506] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:02.300509] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:02.300511] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:02.300514] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:01:02.300518] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=24, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:02.300523] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] will sleep(sleep_us=24000, remain_us=4693809, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=24, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:01:02.304759] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223391541121, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:02.304769] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:02.304776] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:02.304782] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:02.304784] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:02.304791] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.304793] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.304795] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337662304744) [2024-09-26 16:01:02.318027] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:02.318046] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.319757] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337662249048, now=[mts=1727337661320040]) [2024-09-26 16:01:02.319768] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337662249048, now=[mts=1727337661320040], now0=[mts=1727337661320040], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:02.319773] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269014d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337661319671, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:02.319785] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=12][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269014d80}) [2024-09-26 16:01:02.319791] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=6][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269014d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337661319671, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:02.319797] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269014d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337661319671, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337660319277, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269014d80}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:02.319812] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=15][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:02.319816] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:02.319818] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:02.319820] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:02.319824] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:02.319827] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:02.319831] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:01:02.319837] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:02.319840] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:02.319841] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:01:02.319846] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:01:02.319849] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000571) [2024-09-26 16:01:02.319852] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:01:02.319855] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:02.319859] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:01:02.319861] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=1][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:01:02.319864] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:01:02.319869] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:02.319907] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:01:02.319911] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=4][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:01:02.319914] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:01:02.319917] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:02.319921] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:02.319923] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:01:02.320040] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE2-0-0] [lt=3][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:01:02.320049] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:01:02.320051] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:01:02.320060] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:02.320064] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:01:02.320338] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:01:02.320366] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=27][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:02.324833] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:24, local_retry_times:24, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:02.324855] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:02.324862] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:02.324864] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:02.324881] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:02.324908] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:01:02.324911] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:02.347184] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337661849295]}) [2024-09-26 16:01:02.348401] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:01:02.348409] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.350138] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.350149] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337662350135], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:02.350157] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337662350135], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:02.350174] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337662350135]}) [2024-09-26 16:01:02.350179] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:02.350182] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:02.354223] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:02.354324] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:02.354987] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1352651, clean_num=31457) [2024-09-26 16:01:02.378613] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:02.378622] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.391252] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=9] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:02.392242] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=10] current monitor number(seq_=-1) [2024-09-26 16:01:02.395506] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=912224, replace_num=15728) [2024-09-26 16:01:02.395515] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:01:02.404947] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:02.404964] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337662404943) [2024-09-26 16:01:02.404970] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337662204745, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:02.404989] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223391640482, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:02.404996] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:02.405004] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:02.405009] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:02.405012] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:02.405018] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.405020] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.405022] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337662404974) [2024-09-26 16:01:02.408868] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:02.408876] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.430344] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D42BB-0-0] [lt=13][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:02.439130] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:02.439187] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=55][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.448876] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337662350135]}) [2024-09-26 16:01:02.465492] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1727337662465491, bytes: 14712982, bw: 0.003376 MB/s, add_ts: 1000055, add_bytes: 3540 [2024-09-26 16:01:02.466452] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=19] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.466472] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.466476] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.466482] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.466489] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.466495] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.467458] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.467462] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.467474] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=11][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.467480] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.467483] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.467486] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.468407] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.468412] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.468414] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.468453] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.468459] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.468463] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.470400] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.470401] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.470405] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.470405] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.470408] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.470408] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.470416] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:02.470419] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=2][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.471330] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.471344] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.471347] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.471349] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.471347] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.471352] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.472357] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.472358] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.472360] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.472361] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.472362] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.472363] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.473300] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.473304] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.473305] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.473306] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.473307] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.473308] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.474267] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.474272] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.474274] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.474313] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.474318] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.474320] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:02.475246] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.475271] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.476215] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.476223] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.477194] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.477202] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.478157] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.478189] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.479143] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.480111] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.480150] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.481103] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.481106] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.482044] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.482075] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.483063] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.483090] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.484056] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.484060] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.485059] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.485062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.485630] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.485633] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.486964] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.486991] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.487942] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.487992] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.488892] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.488930] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.489894] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.489906] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.490878] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.490925] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.491707] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.491733] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.492840] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.492848] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.493799] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.493830] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.494785] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.494791] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.495756] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.495756] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.496742] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.496744] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.497716] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.497741] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.498685] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.498700] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.499671] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.499663] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.500640] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.500684] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.500696] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:02.500701] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.501614] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.501625] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.502603] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.502632] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.503572] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.503576] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.504556] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.504557] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.505536] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:02.505545] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.505545] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337662505533) [2024-09-26 16:01:02.505550] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.505550] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337662404974, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:02.505569] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223391742013, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:02.505576] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:02.505584] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:02.505589] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:02.505592] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:02.505598] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.505601] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.505603] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337662505553) [2024-09-26 16:01:02.506504] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.506514] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.507492] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.507498] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.508458] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.508461] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.509462] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.509464] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.510431] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.510435] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.512391] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.512391] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.513355] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.513358] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.514317] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.514325] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.515286] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.515346] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.516280] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.516287] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.517274] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:02.526169] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185935-0-0] [lt=28][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:02.526176] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185935-0-0] [lt=7][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:02.526186] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185935-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:02.526192] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185935-0-0] [lt=6][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:02.530904] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:02.530911] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.547514] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6634, mem_used=12478464) [2024-09-26 16:01:02.549501] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337662350135]}) [2024-09-26 16:01:02.555488] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:02.555591] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=66, refresh_score_time=15, wash_time=1) [2024-09-26 16:01:02.556196] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1384108, clean_num=31457) [2024-09-26 16:01:02.561185] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:02.561193] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.566191] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=8] gc stale ls task succ [2024-09-26 16:01:02.591330] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:02.591345] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.596411] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=4] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:01:02.602216] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.602223] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.602224] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.602235] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.606150] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:02.606162] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337662606147) [2024-09-26 16:01:02.606168] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337662505553, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:02.606174] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:02.606185] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:02.606190] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:02.606522] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:02.606535] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:02.606554] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=19][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:02.606558] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:02.606562] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:02.606564] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:02.606567] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:02.606570] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:02.606571] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:02.606573] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:02.606575] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:02.606578] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:02.606581] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:02.606586] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=4][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:02.606593] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:02.606595] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:02.606598] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:02.606602] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:02.606605] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:02.606607] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:02.606610] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:02.606616] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:02.606622] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:02.606624] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:02.606626] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:02.606632] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:02.606636] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:02.606639] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577B-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:02.606642] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:02.606645] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:02.606647] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:02.606650] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337662606445, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:02.606654] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:02.606656] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:02.606687] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:02.606691] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337662606690, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=510, wlock_time=12, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:02.606700] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:02.606703] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:02.606720] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223391842675, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:02.606726] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:02.606732] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:02.606736] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:02.606739] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:02.606745] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.606747] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.606749] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337662606707) [2024-09-26 16:01:02.622746] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14] success to notify_need_writing_throttling True [2024-09-26 16:01:02.622770] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.645028] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337662350135]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:01:02.645043] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:01:02.649974] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337662350135]}) [2024-09-26 16:01:02.653017] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:02.653027] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.664766] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:02.677763] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:02.677774] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:02.677778] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:02.677781] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:02.677785] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:02.677787] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:02.677791] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:02.677793] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:02.677795] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:02.677797] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:02.677799] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:02.677802] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:02.677804] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:02.677806] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:02.677812] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:02.677814] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:02.677817] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:02.677819] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:02.677821] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:01:02.677824] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:02.677827] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:02.677832] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:02.677838] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:02.677840] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:02.677841] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:02.677847] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:02.677850] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD7-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:02.677853] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2458][ServerGTimer][T0][YB42C0A8002D-0006230056A84DD7-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:01:02.677856] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:02.677858] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:02.677860] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:02.677863] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337662677699, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:02.677866] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:02.677896] WDIAG [SHARE] refresh (ob_all_server_tracer.cpp:568) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to get servers_info(ret=-5019, ret="OB_TABLE_NOT_EXIST", GCTX.sql_proxy_=0x55c7486d2e88) [2024-09-26 16:01:02.677898] WDIAG [SHARE] runTimerTask (ob_all_server_tracer.cpp:626) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to refresh all server map(ret=-5019) [2024-09-26 16:01:02.683365] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:02.683373] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.699921] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:02.706728] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:02.706741] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337662706725) [2024-09-26 16:01:02.706748] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337662606171, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:02.706765] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223391942303, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:02.706773] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:02.706792] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:02.706806] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:02.706810] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:02.706817] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.706819] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.706821] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337662706751) [2024-09-26 16:01:02.714574] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:02.714593] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.744832] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:02.744840] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.750706] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337662350135]}) [2024-09-26 16:01:02.756629] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:02.756731] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=53, refresh_score_time=15, wash_time=1) [2024-09-26 16:01:02.757334] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1415565, clean_num=31457) [2024-09-26 16:01:02.775099] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:02.775106] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.785849] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:02.796593] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:01:02.805357] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:01:02.805364] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.807336] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:02.807355] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337662807333) [2024-09-26 16:01:02.807361] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337662706750, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:02.807385] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223392042891, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:02.807392] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:02.807400] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:02.807405] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:02.807408] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:02.807415] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.807417] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.807419] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337662807364) [2024-09-26 16:01:02.821027] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.821056] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.821065] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.821071] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.821076] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.821974] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.821977] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.821985] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.821986] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.821991] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.821993] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:02.835665] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:02.835672] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.851334] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337662350135]}) [2024-09-26 16:01:02.851440] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:02.851450] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337662851438], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:02.851472] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337662851438], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:02.865949] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:02.865957] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.896201] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:02.896209] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.907984] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:02.907999] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337662907981) [2024-09-26 16:01:02.908005] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337662807364, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:02.908021] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223392144468, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:02.908026] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:02.908034] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:02.908039] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:02.908043] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:02.908049] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.908051] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:02.908053] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337662908008) [2024-09-26 16:01:02.920684] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337662920683, bytes: 0, bw: 0.000000 MB/s, add_ts: 1008509, add_bytes: 0 [2024-09-26 16:01:02.926487] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:02.926492] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.951932] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337662851438]}) [2024-09-26 16:01:02.956765] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:02.956773] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.957834] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:02.957934] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:02.958537] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1447022, clean_num=31457) [2024-09-26 16:01:02.965559] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=9][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337662851438]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:01:02.965571] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=11][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:01:02.987045] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:02.987051] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:02.989964] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=9][errcode=-4018] try free send task failed(ret=-4018) [2024-09-26 16:01:02.990027] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:02.996857] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:01:03.008547] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:03.008556] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337663008535) [2024-09-26 16:01:03.008561] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337662908007, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:03.008589] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] get wrs ts(ls_id={id:1}, delta=223392245061, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:03.008595] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:03.008603] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:03.008608] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:03.008611] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:03.008617] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.008619] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.008621] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337663008567) [2024-09-26 16:01:03.017351] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:03.017367] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.024614] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185936-0-0] [lt=37][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:03.024623] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185936-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:03.024633] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185936-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:03.024638] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185936-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:03.038962] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500849, support_ssl=false) [2024-09-26 16:01:03.038999] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221500849, sess.client_addr_="127.0.0.1:36386") [2024-09-26 16:01:03.039006] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f92be35cf90, session_id:3221500849, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:03.039047] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:03.039055] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:03.039095] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA6-0-0] [lt=8] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:03.039277] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA6-0-0] [lt=7][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:03.039286] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA6-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"jS@'BVi0P#[#96C-XTy)"}, ret=-4043) [2024-09-26 16:01:03.039293] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA6-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:03.039296] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA6-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:03.039304] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA6-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:03.039328] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA6-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500849, proxy_sessid:0}) [2024-09-26 16:01:03.039333] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA6-0-0] [lt=4] mark session id unused(sessid=3221500849) [2024-09-26 16:01:03.039339] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA6-0-0] [lt=4][errcode=0] server close connection(sessid=3221500849, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:03.039342] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA6-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500849, proxy_sessid=0) [2024-09-26 16:01:03.039344] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA6-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:03.039348] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA6-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500849, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:03.039383] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f92be35cf90, session_id:3221500849, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337663038910, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:03.039399] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] kill and revert session(conn.sessid_=3221500849, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:03.039403] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] can close safely, do destroy(*s={this:0x7f92be35cf90, session_id:3221500849, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337663038910, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:03.039409] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500849, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:03.047610] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:03.047620] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.052495] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337662851438]}) [2024-09-26 16:01:03.064536] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:03.064548] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:01:03.064553] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:01:03.064574] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=21][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:01:03.064579] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:03.064581] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:03.064584] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:01:03.064587] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:03.064589] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:03.064591] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:03.064595] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:03.064598] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:03.064600] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:03.064603] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:03.064612] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:03.064614] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:03.064617] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:03.064620] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:03.064623] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:01:03.064625] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:03.064628] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:03.064634] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:03.064640] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:03.064642] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:03.064644] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:03.064650] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:03.064653] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:03.064657] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85898-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:01:03.064659] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:03.064662] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85898-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:03.064666] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85898-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:03.064670] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85898-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337663064465, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:03.064673] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85898-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:03.064675] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85898-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:01:03.064697] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:01:03.064699] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85898-0-0] [lt=2][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:01:03.071050] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=9] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:03.071336] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=23] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:03.071346] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:03.071351] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:03.071354] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:03.071357] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:03.071360] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:03.071362] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:03.071365] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:03.071369] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:03.071371] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:03.071374] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:03.071375] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:03.071377] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:03.071379] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:03.071384] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:03.071387] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:03.071389] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:03.071391] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:03.071396] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:01:03.071398] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:03.071400] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:03.071408] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=6][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:03.071413] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:03.071416] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:03.071417] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:03.071422] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:03.071426] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E72-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:03.071429] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2579][RSAsyncTask0][T0][YB42C0A8002D-0006230057584E72-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:01:03.071431] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:03.071434] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:03.071436] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:03.071438] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337663071292, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:03.071442] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:03.071511] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:01:03.077902] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:03.077909] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.091580] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=35] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:03.091557] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:262) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=42] ObArchiveSender is running(thread_index=0) [2024-09-26 16:01:03.108207] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:03.108215] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.109162] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:03.109171] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:03.109177] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337663109159) [2024-09-26 16:01:03.109182] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337663008567, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:03.109188] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:03.109197] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:03.109200] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:03.109489] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:03.109499] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:03.109503] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:03.109506] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:03.109509] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:03.109511] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:03.109514] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:03.109516] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:03.109518] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:03.109555] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=36][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:03.109557] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:03.109559] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:03.109575] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577C-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:03.109603] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337663109602, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=409, wlock_time=11, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:03.109618] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:03.109635] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223392345626, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:03.109640] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:03.109646] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:03.109650] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:03.109653] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:03.109659] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.109661] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.109663] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337663109623) [2024-09-26 16:01:03.138457] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:03.138465] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.138474] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:03.153120] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337662851438]}) [2024-09-26 16:01:03.159068] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:03.159167] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=1) [2024-09-26 16:01:03.159763] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1478479, clean_num=31457) [2024-09-26 16:01:03.162864] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 9392.00) [2024-09-26 16:01:03.162873] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:03.162883] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:03.162887] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:03.169688] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:03.169696] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.197008] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:01:03.199973] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:03.199980] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.209728] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:03.209737] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337663209725) [2024-09-26 16:01:03.209742] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337663109185, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:03.209758] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223392445345, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:03.209763] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:03.209770] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:03.209774] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:03.209777] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:03.209782] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.209784] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.209786] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337663209745) [2024-09-26 16:01:03.211042] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:01:03.211051] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:03.211056] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:01:03.230268] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:03.230277] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.253278] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337662851438]}) [2024-09-26 16:01:03.253305] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D9AE-0-0] [lt=9][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337663253272}) [2024-09-26 16:01:03.259695] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:03.259707] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:03.259712] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:03.259715] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:03.259718] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:03.259720] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:03.259724] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:03.259726] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:03.259728] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:03.259732] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=4][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:03.259735] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:03.259738] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:03.259755] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A7-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:03.260484] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:03.260492] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.290797] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:03.290806] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.309898] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:03.309910] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337663309894) [2024-09-26 16:01:03.309916] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337663209745, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:03.309934] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223392545846, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:03.309940] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:03.309949] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:03.309954] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:03.309957] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:03.309964] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.309966] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.309968] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337663309920) [2024-09-26 16:01:03.319903] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD8-0-0] [lt=2][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-09-26 16:01:03.319913] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD8-0-0] [lt=10][errcode=-4002] refresh sever list failed(ret=-4002) [2024-09-26 16:01:03.319916] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD8-0-0] [lt=3][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-09-26 16:01:03.320446] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337664250069, now=[mts=1727337662320337]) [2024-09-26 16:01:03.320467] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=21][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337664250069, now=[mts=1727337662320337], now0=[mts=1727337662320337], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:03.320473] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92690275c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337662319749, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:03.320490] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92690275c0}) [2024-09-26 16:01:03.320498] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92690275c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337662319749, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:03.320520] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=20] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92690275c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337662319749, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337662320311, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92690275c0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:03.320540] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=19][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:03.320545] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:03.320547] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=1][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:03.320549] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:03.320553] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:03.320557] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:03.320562] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=999507, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337664320069) [2024-09-26 16:01:03.320566] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:03.320573] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:03.320575] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:03.320577] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:03.320582] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:03.320621] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:01:03.320624] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:03.321078] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:01:03.321087] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.324038] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337662324907]) [2024-09-26 16:01:03.324044] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337662324907], now0=[mts=1727337662324907], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:03.324049] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337662323859, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:03.324064] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:01:03.324071] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337662323859, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:03.324077] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337662323859, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:03.324092] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:03.324096] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:03.324100] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:03.324102] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:03.324104] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:01:03.324108] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=25, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:03.324112] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=25000, remain_us=3670221, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=25, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:01:03.349398] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:25, local_retry_times:25, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:03.349415] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:03.349419] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:03.349420] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:03.349425] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:03.349456] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:01:03.349459] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:03.350925] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:53, tid:2919}]) [2024-09-26 16:01:03.351180] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:03.351187] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.352478] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.352489] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337663352475], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:03.352496] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337663352475], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:03.352500] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:03.352504] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:03.354361] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337663352475]}) [2024-09-26 16:01:03.360349] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:03.360454] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=52, refresh_score_time=16, wash_time=1) [2024-09-26 16:01:03.361069] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1509936, clean_num=31457) [2024-09-26 16:01:03.381309] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:03.381323] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.391380] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=9] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:03.393354] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=27] current monitor number(seq_=-1) [2024-09-26 16:01:03.397265] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:01:03.406102] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] ====== tenant freeze timer task ====== [2024-09-26 16:01:03.406113] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:01:03.406146] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337660633362, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337660633318, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337660633383, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:01:03.406172] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:01:03.406240] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [2996][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E4-0-0] [lt=3] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:01:03.406249] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [2996][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E4-0-0] [lt=7] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:01:03.406253] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [2996][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E4-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:01:03.406256] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [2996][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E4-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:01:03.406260] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [2996][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E4-0-0] [lt=4][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:01:03.406263] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [2996][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E4-0-0] [lt=3][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:01:03.406265] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [2996][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E4-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:01:03.406274] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [2996][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E4-0-0] [lt=9] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:01:03.406276] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [2996][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E4-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:03.406278] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [2996][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E4-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:03.406281] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [2996][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E4-0-0] [lt=1][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:03.406308] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851E4-0-0] [lt=3] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:01:03.410953] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:03.410968] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337663410950) [2024-09-26 16:01:03.410973] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337663309919, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:03.411000] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223392647428, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:03.411007] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:03.411015] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:03.411019] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:03.411022] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:03.411029] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.411032] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.411033] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337663410976) [2024-09-26 16:01:03.411561] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:01:03.411570] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.429550] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D4330-0-0] [lt=7][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:03.442485] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:03.442491] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.454912] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337663352475]}) [2024-09-26 16:01:03.466601] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.466608] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.466613] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.466630] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.466633] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.466635] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.467652] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.467660] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.467665] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.467680] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.467682] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.467684] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.468554] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.468572] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.468576] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.468591] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.468594] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.468596] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.469598] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.469608] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.469614] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.469632] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.469636] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.469639] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.470508] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.470512] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.470514] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.470522] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.470524] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.470525] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.471071] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.471074] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.471076] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.471085] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.471086] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.471088] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.472465] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.472468] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.472470] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.472467] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=19] PNIO [ratelimit] time: 1727337663472466, bytes: 14717239, bw: 0.004032 MB/s, add_ts: 1006975, add_bytes: 4257 [2024-09-26 16:01:03.472476] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.472478] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.472498] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=19][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.473102] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.473105] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.473107] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.473112] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.473114] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.473113] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:03.473116] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:03.473118] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.474414] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.474433] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.475398] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.475405] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.475939] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.475947] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.477340] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.477347] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.477891] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.477898] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.478554] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.478561] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.479347] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.479357] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.480277] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.480292] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.481249] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.481296] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.481839] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.481851] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=33][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.482491] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.482505] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.484217] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.484229] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.486165] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.486181] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.487116] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.487131] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.488118] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.488143] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.489074] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.489078] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.491131] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.491139] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.492053] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.492060] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.492967] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.492979] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.493950] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.493955] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.494932] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.494938] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.495954] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.495962] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.496898] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.496904] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.497460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.497465] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.498177] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.498182] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.498762] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.498766] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.499831] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.499836] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.500796] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.500804] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.501502] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.501507] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.502766] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.502772] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.503717] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:03.503719] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.503725] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.503733] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.504697] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.504702] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.505678] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.505683] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.506679] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.506696] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.507583] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.507589] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.508609] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.508614] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.509585] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.509598] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.510209] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.510213] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.511582] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.511590] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.511588] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:03.511595] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337663511585) [2024-09-26 16:01:03.511600] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337663410976, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:03.511619] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223392748052, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:03.511627] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:03.511634] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:03.511641] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:03.511644] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:03.511650] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.511652] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.511654] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337663511605) [2024-09-26 16:01:03.512276] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:03.512281] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4038] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:03.513473] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:03.522521] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185937-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:03.522530] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185937-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:03.522538] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185937-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:03.522543] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185937-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:03.528112] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:218) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-09-26 16:01:03.528124] INFO [COMMON] handle (memory_dump.cpp:498) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-09-26 16:01:03.528150] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3] update_check_range(min_check_version=343, max_check_version=344, global_version=345) [2024-09-26 16:01:03.530604] INFO handle (memory_dump.cpp:626) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=4] statistics: tenant_cnt: 3, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 80 198 0 1 2 362 1 0 1 4 1 1 0 1 5 1 5 0 1 7 1 2 0 1 8 60 1 0 1 12 4 2 0 1 13 2 1 0 1 14 1 1 0 1 18 54 3 0 500 0 52 216 0 500 7 2 3 0 500 8 42 1 0 500 9 2 1 0 500 10 6 2 0 500 16 1 1 0 500 17 6 7 0 500 22 3 49 0 500 23 22 10 0 508 0 2 8 0 508 8 3 1 0 cost_time: 2457 [2024-09-26 16:01:03.535027] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:03.535043] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.541838] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=15] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:01:03.547675] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=19] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6642, mem_used=12478464) [2024-09-26 16:01:03.551604] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4038, dropped:5, tid:2546}]) [2024-09-26 16:01:03.555563] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337663352475]}) [2024-09-26 16:01:03.561461] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:03.561581] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=51, refresh_score_time=30, wash_time=2) [2024-09-26 16:01:03.562193] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1541393, clean_num=31457) [2024-09-26 16:01:03.565255] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:03.565264] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.566256] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=6] gc stale ls task succ [2024-09-26 16:01:03.591970] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] LogLoopThread round_cost_time(us)(round_cost_time=8) [2024-09-26 16:01:03.595562] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:03.595568] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.597568] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:01:03.601606] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.602205] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.602213] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.602657] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.605677] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:03.605691] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:03.605695] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:03.605699] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:03.605703] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:03.605705] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:03.605709] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:03.605711] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:03.605714] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:03.605716] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:03.605717] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:03.605720] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:03.605722] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:03.605724] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:03.605746] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:03.605750] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:03.605754] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:03.605756] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:03.605760] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, ret=-5019) [2024-09-26 16:01:03.605763] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:03.605766] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:03.605775] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=7][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:03.605781] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:03.605784] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:03.605786] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:03.605792] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:03.605796] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:03.605799] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD8-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-09-26 16:01:03.605802] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:01:03.605806] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:03.605810] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:03.605813] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337663605605, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:01:03.605816] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:03.605819] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f92b4056738, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, columns_str="zone") [2024-09-26 16:01:03.605825] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:01:03.605859] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:634) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f92b4056738, where_condition="where svr_ip='192.168.0.45' and svr_port=2882", zone_name_holder=) [2024-09-26 16:01:03.605862] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:567) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:01:03.605866] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:576) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:01:03.605868] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:03.605874] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:03.612133] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:03.612154] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337663612130) [2024-09-26 16:01:03.612159] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337663511604, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:03.612164] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:03.612173] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:03.612175] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:03.612495] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:03.612503] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:03.612506] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:03.612509] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:03.612512] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:03.612514] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:03.612517] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:03.612519] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:03.612520] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:03.612522] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:03.612524] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:03.612525] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:03.612527] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:03.612529] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:03.612534] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:03.612536] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:03.612538] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:03.612540] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:03.612542] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:03.612545] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:03.612547] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:03.612551] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:03.612555] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:03.612557] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:03.612558] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:03.612563] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:03.612566] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:03.612569] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577D-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:03.612571] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:03.612573] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:03.612575] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:03.612577] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337663612422, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:03.612580] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:03.612582] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:03.612603] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:03.612606] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337663612605, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=437, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:03.612612] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:03.612614] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:03.612631] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223392848578, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:03.612636] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:03.612642] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:03.612646] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:03.612648] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:03.612653] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.612655] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.612659] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337663612617) [2024-09-26 16:01:03.625809] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:03.625825] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.649305] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855F5-0-0] [lt=4] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:03.649336] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855F7-0-0] [lt=7] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:03.655745] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337663352475]}) [2024-09-26 16:01:03.656041] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:03.656068] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.664839] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:03.686234] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:03.686246] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.712473] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:03.712486] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337663712469) [2024-09-26 16:01:03.712492] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337663612162, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:03.712510] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223392948456, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:03.712516] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:03.712524] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:03.712529] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:03.712531] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:03.712537] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.712539] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.712541] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337663712496) [2024-09-26 16:01:03.716688] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:03.716707] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.739112] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=7] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:01:03.746901] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:03.746911] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.755765] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337663352475]}) [2024-09-26 16:01:03.762666] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:03.762777] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:03.762788] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1572850, clean_num=31457) [2024-09-26 16:01:03.778187] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:03.778203] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.792539] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:03.797833] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:01:03.808415] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:03.808425] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.812755] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:03.812769] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337663812751) [2024-09-26 16:01:03.812775] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337663712495, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:03.812795] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223393047878, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:03.812802] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:03.812810] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:03.812815] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:03.812818] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:03.812824] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.812826] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.812828] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337663812779) [2024-09-26 16:01:03.820499] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.820514] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.820520] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=4] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.820524] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.820529] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.820787] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.822127] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.822131] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=5] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.822135] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.822139] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.822146] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:03.839710] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:01:03.839719] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.853006] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:03.853015] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337663853003], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:03.853022] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337663853003], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:03.856371] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337663853003]}) [2024-09-26 16:01:03.869981] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:03.869991] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.900523] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:03.900544] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.912958] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:03.912973] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337663912955) [2024-09-26 16:01:03.912979] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337663812779, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:03.913017] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223393149432, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:03.913024] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:03.913031] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:03.913036] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:03.913038] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:03.913045] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.913047] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:03.913049] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337663912995) [2024-09-26 16:01:03.927607] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337663927607, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006924, add_bytes: 0 [2024-09-26 16:01:03.931510] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:03.931521] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.956962] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337663853003]}) [2024-09-26 16:01:03.961790] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:03.961799] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.961812] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] BlockGCTimerTask success(ret=0, cost_time_us=24, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.963261] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:03.963404] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=52, refresh_score_time=15, wash_time=2) [2024-09-26 16:01:03.964041] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=0, clean_num=31457) [2024-09-26 16:01:03.964397] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-09-26 16:01:03.964409] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-09-26 16:01:03.964413] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-09-26 16:01:03.964431] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-09-26 16:01:03.964435] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:03.964438] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:03.964441] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-09-26 16:01:03.964443] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:03.964446] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:03.964448] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:03.964451] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:03.964453] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:03.964467] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:03.964470] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:03.964477] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:03.964480] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:03.964483] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:03.964486] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:03.964491] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:01:03.964494] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:03.964497] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:03.964504] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:03.964509] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:03.964512] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:03.964514] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:03.964519] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:03.964524] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:03.964527] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2024-09-26 16:01:03.964530] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:01:03.964533] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:03.964537] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:03.964540] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337663964313, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:01:03.964543] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:03.964547] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1111) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:01:03.964578] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:801) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:01:03.964582] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:67) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:03.964586] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:436) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=4][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:03.964589] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:95) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:03.964590] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:109) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=1] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:03.964834] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-09-26 16:01:03.964844] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2024-09-26 16:01:03.964848] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2024-09-26 16:01:03.964850] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2024-09-26 16:01:03.964852] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:03.964855] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:03.964857] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_tenant' doesn't exist [2024-09-26 16:01:03.964859] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:03.964861] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:03.964863] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:03.964864] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:03.964868] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=4][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:03.964870] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:03.964872] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:03.964876] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:03.964877] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:03.964879] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:03.964881] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:03.964882] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:01:03.964884] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:03.964886] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:03.964890] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:03.964894] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:03.964895] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:03.964898] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:03.964903] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:03.964906] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:03.964908] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, aret=-5019, ret=-5019) [2024-09-26 16:01:03.964910] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:01:03.964913] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:03.964914] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:03.964916] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337663964808, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:01:03.964919] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:03.964921] WDIAG [SHARE] read_tenants (ob_unit_table_operator.cpp:1174) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=1][errcode=-5019] execute sql failed(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:01:03.964937] WDIAG [SHARE] get_tenants (ob_unit_table_operator.cpp:108) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] fail read tenants(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:01:03.964940] WDIAG [SHARE] get_tenants (ob_unit_getter.cpp:198) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] ut_operator get_resource_pools failed(ret=-5019) [2024-09-26 16:01:03.964942] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:125) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=2][errcode=-5019] get cluster tenants fail(ret=-5019) [2024-09-26 16:01:03.964954] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:325) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=4] cal_all_part_disk_default_percentage succ(data_dir="/root/observer/store/sstable", clog_dir="/root/observer/store/clog", shared_mode=true, data_disk_total_size=99765694464, data_disk_default_percentage=60, clog_disk_total_size=99765694464, clog_disk_default_percentage=30) [2024-09-26 16:01:03.964958] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:361) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=4] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=99765694464, disk_size=15032385536) [2024-09-26 16:01:03.964961] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=3] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=99765694464) [2024-09-26 16:01:03.964968] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=6] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=99765694464, this={dir::"/root/observer/store/clog/log_pool", dir_fd:15, meta_fd:16, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:9388, max_block_id:9581, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2024-09-26 16:01:03.964974] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:133) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DD9-0-0] [lt=6] refresh tenant config(tenants=[], ret=-5019) [2024-09-26 16:01:03.990162] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:03.992088] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5] success to notify_need_writing_throttling True [2024-09-26 16:01:03.992096] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:03.997993] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:01:04.013097] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:04.013107] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337664013095) [2024-09-26 16:01:04.013113] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337663912982, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:04.013130] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223393249112, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:04.013135] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:04.013143] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:04.013148] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:04.013150] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:04.013156] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.013158] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.013159] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337664013116) [2024-09-26 16:01:04.022401] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:04.022408] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.023182] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185938-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:04.023193] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185938-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:04.023202] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185938-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:04.023207] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185938-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:04.040138] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500854, support_ssl=false) [2024-09-26 16:01:04.040180] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221500854, sess.client_addr_="127.0.0.1:36388") [2024-09-26 16:01:04.040187] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500854, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:04.040227] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:04.040237] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=5] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:04.040275] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA7-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:04.040464] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA7-0-0] [lt=5][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:04.040470] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA7-0-0] [lt=5][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"\sPrpZTtj:"97HD\$;client_type_=3, conn->client_version_=0) [2024-09-26 16:01:04.040564] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500854, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337664039925, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:04.040571] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] kill and revert session(conn.sessid_=3221500854, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:04.040574] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500854, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337664039925, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:04.040578] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500854, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:04.053595] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:04.053603] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.057544] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337663853003]}) [2024-09-26 16:01:04.065638] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:04.065649] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:01:04.065654] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:01:04.065670] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:01:04.065675] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:04.065678] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:04.065681] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:01:04.065684] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:04.065686] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:04.065688] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:04.065690] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:04.065695] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:04.065698] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:04.065700] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:04.065706] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:04.065709] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:04.065712] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:04.065714] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:04.065716] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:01:04.065719] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:04.065722] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:04.065727] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:04.065733] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:04.065736] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:04.065737] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:04.065743] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:04.065747] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C85899-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:04.065750] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:01:04.065753] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:04.065755] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:04.065760] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85899-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:04.065763] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337664065574, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:04.065766] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85899-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:04.065768] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:01:04.065791] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85899-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:01:04.065794] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C85899-0-0] [lt=2][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:01:04.072198] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=27] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:04.072458] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:04.072482] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=23][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:04.072486] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:04.072490] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:04.072495] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=4][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:04.072497] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:04.072500] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:04.072503] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:04.072504] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:04.072507] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=3][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:04.072511] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:04.072513] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:04.072516] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:04.072517] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:04.072530] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D88-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:04.083880] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=31] success to notify_need_writing_throttling True [2024-09-26 16:01:04.083889] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.091715] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:04.113173] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:04.113182] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:04.113194] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337664113170) [2024-09-26 16:01:04.113200] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337664013116, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:04.113207] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:04.113217] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:04.113233] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:04.113608] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577E-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:04.113618] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577E-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:04.113643] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577E-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:04.113673] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337664113672, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=459, wlock_time=24, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:04.113685] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:04.113703] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223393349674, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:04.113711] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:04.113718] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:04.113724] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:04.113727] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:04.113734] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.113737] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.113741] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337664113690) [2024-09-26 16:01:04.114204] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.114212] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.144505] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:04.144513] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.144521] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:04.156153] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-09-26 16:01:04.156466] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD8-0-0] [lt=8] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-09-26 16:01:04.156477] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD8-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:04.156515] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD8-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:04.158115] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337663853003]}) [2024-09-26 16:01:04.164986] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 12222.00) [2024-09-26 16:01:04.165001] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:04.165013] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:04.165017] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:04.165088] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:04.165194] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=54, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:04.165812] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=31457, clean_num=31457) [2024-09-26 16:01:04.165940] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:520) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] [CACHE] tenant_id= 1 cache memory info: [CACHE] tenant_id= 1 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 28 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_block_cache | cache_size= 8321024 | cache_store_size= 4161536 | cache_map_size= 4159488 | kv_cnt= 20 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 393 | hold_size= 0 [2024-09-26 16:01:04.165949] INFO print_current_status (ob_kvcache_hazard_version.cpp:431) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] [KVCACHE-HAZARD] hazard version status info: current version: 16612 | min_version= 16612 | total thread store count: 62 | total nodes count: 1 | [KVCACHE-HAZARD] i= 56 | thread_id= 2513 | inited= 1 | waiting_nodes_count= 1 | last_retire_version= 16611 | acquired_version=18446744073709551615 | [2024-09-26 16:01:04.174810] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:01:04.174819] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.198162] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:01:04.202057] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:01:04.202064] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=5] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:01:04.202071] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:01:04.205042] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:04.205047] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.213191] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223393449553, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:04.213200] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:04.213207] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:04.213212] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:04.213215] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:04.213220] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.213222] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.213224] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337664213179) [2024-09-26 16:01:04.235281] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.235300] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.259787] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337663853003]}) [2024-09-26 16:01:04.259804] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D9BA-0-0] [lt=8][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337664259781}) [2024-09-26 16:01:04.260205] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A8-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:04.260216] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A8-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:04.260242] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A8-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:04.266699] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.266708] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.297781] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.297791] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.313464] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:04.313475] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337664313461) [2024-09-26 16:01:04.313480] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337664113204, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:04.313497] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223393549923, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:04.313502] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:04.313509] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:04.313513] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:04.313516] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:04.313521] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.313523] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.313525] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337664313484) [2024-09-26 16:01:04.321228] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337664250069, now=[mts=1727337663320620]) [2024-09-26 16:01:04.321239] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337664250069, now=[mts=1727337663320620], now0=[mts=1727337663320620], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:04.321246] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92690275c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337663320432, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:04.321283] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=36][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92690275c0}) [2024-09-26 16:01:04.321300] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=17][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92690275c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337663320432, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:04.321309] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=8] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92690275c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337663320432, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337662320311, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92690275c0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:04.321327] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=18][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:04.321333] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:04.321335] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:04.321337] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:04.321341] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:04.321347] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=5][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:04.321351] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:01:04.321358] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:04.321360] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:04.321362] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:01:04.321368] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:01:04.321372] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=3] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2001060) [2024-09-26 16:01:04.321375] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:01:04.321378] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:04.321383] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:01:04.321385] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:01:04.321388] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:01:04.321393] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:04.321418] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:01:04.321421] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:01:04.321424] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:01:04.321426] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:04.321430] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=4][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:04.321433] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:01:04.321558] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE3-0-0] [lt=3][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:01:04.321567] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:01:04.321570] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:01:04.321581] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:04.321590] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=8][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:01:04.321853] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:01:04.321862] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=8][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:04.329091] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:04.329111] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.350580] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337663349455]) [2024-09-26 16:01:04.350591] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337663349455], now0=[mts=1727337663349455], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:04.350598] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337663348497, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:04.350612] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:01:04.350618] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337663348497, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:04.350626] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337663348497, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:04.350643] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:04.350648] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:04.350650] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:04.350655] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:04.350658] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:01:04.350661] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=26, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:04.350665] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] will sleep(sleep_us=26000, remain_us=2643667, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=26, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:01:04.353667] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.353675] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337664353665], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:04.353691] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337664353665], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:04.353696] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:04.353699] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=2] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:04.360361] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337664353665]}) [2024-09-26 16:01:04.360364] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.360372] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.367354] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=2] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:04.367519] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] Wash time detail, (compute_wash_size_time=66, refresh_score_time=22, wash_time=2) [2024-09-26 16:01:04.368180] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=62914, clean_num=31457) [2024-09-26 16:01:04.376905] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:26, local_retry_times:26, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:04.376939] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=30][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:04.376954] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:04.376956] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:04.376963] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:04.377003] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:01:04.377008] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:04.390574] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.390585] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.391449] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=12] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:04.392495] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [2645][Blacklist][T0][Y0-0000000000000000-0-0] [lt=6] blacklist_loop exec finished(cost_time=9, is_enabled=true, send_cnt=0) [2024-09-26 16:01:04.393494] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=9] current monitor number(seq_=-1) [2024-09-26 16:01:04.398181] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:01:04.414021] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:04.414041] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337664414017) [2024-09-26 16:01:04.414047] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337664313483, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:04.414067] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223393649678, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:04.414073] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:04.414082] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:04.414088] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:04.414091] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:04.414097] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.414100] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.414102] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337664414050) [2024-09-26 16:01:04.421824] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:04.421835] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.425049] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F55-0-0] [lt=1] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:01:04.425081] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F55-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:04.425092] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F55-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:04.425099] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F55-0-0] [lt=0] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:01:04.425127] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F55-0-0] [lt=0] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:04.434555] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D4378-0-0] [lt=21][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:04.452098] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:04.452118] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.460915] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337664353665]}) [2024-09-26 16:01:04.467745] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.467756] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.467762] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.467762] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.467769] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.467773] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.468713] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.468715] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.468721] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.468722] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.468725] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.468726] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.469685] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.469685] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.469688] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.469690] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.469690] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.469693] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.471649] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.471644] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.471660] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.471663] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.471656] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.471676] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=20][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.472613] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1727337664472612, bytes: 14720344, bw: 0.002961 MB/s, add_ts: 1000146, add_bytes: 3105 [2024-09-26 16:01:04.472663] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.472663] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=13] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.472666] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.472668] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.472668] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.472670] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.473601] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.473610] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.473615] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.474572] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.474578] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.474578] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.474581] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.474581] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.474583] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.475560] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.475565] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.475568] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.475562] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.475585] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=23][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.475587] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.476514] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.476517] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.476519] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:04.476527] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.477540] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.477540] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.478478] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.478485] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.479467] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.479531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.480461] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.480468] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.481395] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.481396] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.482368] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.482405] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.482409] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:04.482415] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.483380] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.483418] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.484368] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.484371] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.486329] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.486351] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.487264] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.487311] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=18][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.488281] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.488285] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.488869] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.488874] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.490215] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.490216] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.491161] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.491161] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.492200] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.492251] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.493169] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.493176] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.494140] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.494153] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.495134] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.495183] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.496076] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.496070] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.497030] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.497062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.498020] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.498021] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.498998] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.499040] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.499977] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.499979] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.500941] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.500988] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.502905] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.502904] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.503867] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=29][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.503900] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.504839] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.504841] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.505822] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.505859] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.506810] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.506825] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.507779] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.507784] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.508755] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.508798] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.509748] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.509775] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.510703] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.511698] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.511699] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.512649] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:04.512659] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.512706] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.512708] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=26][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.513640] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.513704] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.514619] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:04.514631] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337664514615) [2024-09-26 16:01:04.514636] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337664414050, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:04.514654] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223393750204, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:04.514660] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:04.514667] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:04.514672] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:04.514672] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.514675] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:04.514680] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.514682] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.514684] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337664514639) [2024-09-26 16:01:04.514691] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.515577] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.515577] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.516554] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.516604] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.517536] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.517543] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.518529] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.518546] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.519498] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.519506] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:04.520632] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185939-0-0] [lt=40][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:04.520640] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185939-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:04.520650] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185939-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:04.520655] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185939-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:04.542952] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.542972] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.548901] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=1, size_used=6654, mem_used=12478464) [2024-09-26 16:01:04.561513] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337664353665]}) [2024-09-26 16:01:04.566422] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=8] gc stale ls task succ [2024-09-26 16:01:04.568472] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:04.568587] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=59, refresh_score_time=28, wash_time=2) [2024-09-26 16:01:04.569194] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=94371, clean_num=31457) [2024-09-26 16:01:04.573192] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:04.573201] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.598880] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=927952, replace_num=15728) [2024-09-26 16:01:04.598890] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:01:04.602566] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.602566] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.602569] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.602577] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.603512] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.603518] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.609396] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:04.616216] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:04.616227] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337664616184) [2024-09-26 16:01:04.616254] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337664514639, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:04.616261] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:04.616269] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:04.616273] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:04.616624] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577F-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:04.616659] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788577F-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:04.616693] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337664616692, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=426, wlock_time=8, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:04.616702] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:04.616718] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223393852694, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:04.616723] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:04.616730] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:04.616735] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:04.616738] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:04.616743] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.616745] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.616747] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337664616706) [2024-09-26 16:01:04.633770] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:01:04.633779] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.657239] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:189, tid:2736}]) [2024-09-26 16:01:04.662122] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337664353665]}) [2024-09-26 16:01:04.664083] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.664102] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.665026] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:04.694168] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.694179] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.717777] WDIAG [PALF] convert_to_ts (scn.cpp:265) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2024-09-26 16:01:04.717786] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1727337664717776, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1727337664717776, force_self_check=false, weak_read_refresh_interval=100000) [2024-09-26 16:01:04.717802] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:04.717809] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337664717800) [2024-09-26 16:01:04.717813] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337664616258, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:04.717832] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223393954265, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:04.717837] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:04.717847] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:04.717852] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:04.717855] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:04.717860] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.717863] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.717865] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337664717817) [2024-09-26 16:01:04.724596] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:04.724606] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.754727] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.754743] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.761709] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=11] XA scheduler heartbeat task statistics(avg_time=6) [2024-09-26 16:01:04.762725] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337664353665]}) [2024-09-26 16:01:04.769627] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:04.769735] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=54, refresh_score_time=17, wash_time=1) [2024-09-26 16:01:04.770347] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=125828, clean_num=31457) [2024-09-26 16:01:04.784361] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=3] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-09-26 16:01:04.784373] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=9] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-09-26 16:01:04.784380] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=5] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=693) [2024-09-26 16:01:04.785153] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:04.785160] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.797879] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:04.799815] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:01:04.815461] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:04.815471] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.818440] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:04.818449] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337664818436) [2024-09-26 16:01:04.818466] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337664717816, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:04.818481] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get_max_applied_scn(scn={val:1727114270763524000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:374, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:1727114270763524000, v:0}, max_applied_cb_scn_:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:04.818490] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] get_min_unreplayed_log_info(lsn={lsn:629872191939}, scn={val:1727114270763524001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:01:04.818509] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=223394054910, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:04.818516] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:04.818524] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:04.818528] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:04.818531] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:04.818537] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.818540] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.818542] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337664818470) [2024-09-26 16:01:04.821368] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.821380] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.821371] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.821384] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.821389] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.821391] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.821393] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.821398] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.821404] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.822271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.822279] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:04.845738] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:01:04.845744] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.854676] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:04.854683] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337664854674], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:04.854689] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337664854674], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:04.863318] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337664854674]}) [2024-09-26 16:01:04.876007] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:04.876015] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.876333] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=5] table not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, table_name.ptr()="data_size:16, data:5F5F616C6C5F6D657267655F696E666F", ret=-5019) [2024-09-26 16:01:04.876346] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, ret=-5019) [2024-09-26 16:01:04.876350] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_merge_info, db_name=oceanbase) [2024-09-26 16:01:04.876364] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_merge_info) [2024-09-26 16:01:04.876367] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:04.876371] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:04.876374] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_merge_info' doesn't exist [2024-09-26 16:01:04.876377] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:04.876379] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:04.876381] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:04.876384] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:04.876387] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:04.876389] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:04.876392] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:04.876399] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:04.876402] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:04.876405] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:04.876408] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:04.876411] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_merge_info WHERE tenant_id = '1', ret=-5019) [2024-09-26 16:01:04.876413] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:04.876416] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:04.876422] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:04.876428] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:04.876431] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:04.876435] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:04.876442] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:04.876459] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:04.876463] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508C-0-0] [lt=4][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, aret=-5019, ret=-5019) [2024-09-26 16:01:04.876468] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:01:04.876471] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:04.876473] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:04.876476] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337664876267, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:01:04.876480] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:04.876483] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:49) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:01:04.876513] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:890) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] failed to load global merge info(ret=-5019, ret="OB_TABLE_NOT_EXIST", global_merge_info={tenant_id:1, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2024-09-26 16:01:04.876527] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1005) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] fail to refresh merge info(tmp_ret=-5019, tmp_ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:04.876759] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:01:04.876765] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:04.906268] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.906276] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.918987] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:04.918999] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337664918985) [2024-09-26 16:01:04.919004] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337664818470, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:04.919021] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223394155452, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:04.919027] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:04.919035] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:04.919040] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:04.919045] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:04.919050] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.919052] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:04.919054] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337664919007) [2024-09-26 16:01:04.920937] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] ====== checkpoint timer task ====== [2024-09-26 16:01:04.921345] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x7f92a6a04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1726077600526835000, v:0}, last_push_gc_task_ts_:1727334198745636, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:01:04.921360] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=15] tx ctx memtable get rec scn(this={ObITable:{this:0x7f92a22a4080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f92a22a4080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:01:04.921373] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1723174350926847000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-09-26 16:01:04.921388] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:01:04.921473] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=82] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}, min_rec_scn={val:1726163586617842000, v:0}, ls_id_={id:1}) [2024-09-26 16:01:04.921486] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:01:04.921489] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:01:04.921492] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:01:04.921498] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=1] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1726163586617842000, v:0}, checkpoint_scn_in_ls_meta={val:1726163586617842000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-09-26 16:01:04.921506] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}, curr_base_lsn={lsn:627966418944}, new_base_lsn={lsn:627966418944}, lsn={lsn:627966418944}) [2024-09-26 16:01:04.921511] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=5] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:627966418944}, ls->get_ls_id()={id:1}) [2024-09-26 16:01:04.921514] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-09-26 16:01:04.935584] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337664935584, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007977, add_bytes: 0 [2024-09-26 16:01:04.936542] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:04.936550] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.963901] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337664854674]}) [2024-09-26 16:01:04.966857] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:04.966864] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:04.970892] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:04.970995] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=49, refresh_score_time=15, wash_time=1) [2024-09-26 16:01:04.971597] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=157285, clean_num=31457) [2024-09-26 16:01:04.983459] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337664854674]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:01:04.983470] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=11][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:01:04.990298] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-09-26 16:01:04.990317] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=17] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:01:04.990325] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=7] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:01:04.990329] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=3] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:01:04.991256] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:04.991961] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] Dumping All Cache Objs(alloc_obj_list.count()=3, alloc_obj_list=[{obj_id:78, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157779}, {obj_id:79, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:5, added_to_lc:true, mem_used:175172}]) [2024-09-26 16:01:04.991990] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=28] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:01:04.997620] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:04.997632] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.000142] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:01:05.019026] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:05.019042] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337665019022) [2024-09-26 16:01:05.019048] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337664919007, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:05.019072] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223394254959, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.019078] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:05.019087] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:05.019092] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:05.019095] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:05.019124] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.019126] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.019128] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337665019051) [2024-09-26 16:01:05.026741] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:05.026754] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-09-26 16:01:05.026759] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:05.026762] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-09-26 16:01:05.026766] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:05.026768] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:05.026777] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-09-26 16:01:05.026779] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:05.026781] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:05.026783] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:05.026786] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=3][errcode=-5019] resolve table failed(ret=-5019) [2024-09-26 16:01:05.026790] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:2822) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=4][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-09-26 16:01:05.026793] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2676) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2][errcode=-5019] resolve joined table failed(ret=-5019) [2024-09-26 16:01:05.026802] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:05.026804] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:05.026807] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:05.026809] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:05.026817] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:05.026819] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:05.026823] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:05.026826] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:05.026831] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-09-26 16:01:05.026834] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:05.026837] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:05.026844] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:05.026850] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:05.026853] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:05.026854] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:05.026860] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:05.026864] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA7-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:05.026868] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [3024][BlackListServic][T0][YB42C0A8002D-0006230056284FA7-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-09-26 16:01:05.026871] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:01:05.026874] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:05.026878] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:05.026881] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337665026626, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:01:05.026884] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:05.026887] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:222) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:01:05.026915] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3] ls blacklist refresh finish(cost_time=562) [2024-09-26 16:01:05.028374] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:05.028391] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.029585] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593A-0-0] [lt=38][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:05.029598] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593A-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:05.029606] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593A-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:05.029612] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593A-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:05.041229] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] new mysql sessid created(conn.sessid_=3221500861, support_ssl=false) [2024-09-26 16:01:05.041257] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221500861, sess.client_addr_="127.0.0.1:36390") [2024-09-26 16:01:05.041264] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500861, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:05.041284] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:05.041292] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:05.041295] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:280) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=3] mysql login queue(mysql_queue->queue_.size()=1) [2024-09-26 16:01:05.041345] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA7-0-0] [lt=21] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:05.041535] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA7-0-0] [lt=5][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:05.041542] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA7-0-0] [lt=6][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"3,$lu(d@S`jq{67Ruwtb"}, ret=-4043) [2024-09-26 16:01:05.041548] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA7-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:05.041552] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA7-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:05.041559] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA7-0-0] [lt=1] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:05.041578] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA7-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500861, proxy_sessid:0}) [2024-09-26 16:01:05.041582] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA7-0-0] [lt=3] mark session id unused(sessid=3221500861) [2024-09-26 16:01:05.041585] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA7-0-0] [lt=1][errcode=0] server close connection(sessid=3221500861, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:05.041588] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA7-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500861, proxy_sessid=0) [2024-09-26 16:01:05.041590] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA7-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:05.041592] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA7-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500861, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:05.041620] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500861, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337665041130, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:05.041638] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500861, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:05.041642] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=4] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500861, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337665041130, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:05.041646] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500861, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:05.058623] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:05.058631] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.064546] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337664854674]}) [2024-09-26 16:01:05.066757] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:05.066770] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:01:05.066774] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:01:05.066777] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:01:05.066780] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:05.066782] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:05.066785] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:01:05.066788] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=3][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:05.066790] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:05.066792] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:05.066793] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:05.066796] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:05.066798] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:05.066800] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:05.066818] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:05.066821] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:05.066824] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:05.066826] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:05.066828] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:01:05.066835] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=6][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:05.066838] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:05.066843] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:05.066849] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:05.066853] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:05.066855] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:05.066860] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:05.066863] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:05.066867] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589A-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:01:05.066870] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:05.066873] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:05.066877] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589A-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:05.066879] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337665066695, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:05.066883] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589A-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:05.066885] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589A-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:01:05.066904] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589A-0-0] [lt=2][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:01:05.066908] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589A-0-0] [lt=4][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:01:05.073290] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:05.073570] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:05.073579] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:05.073583] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:05.073586] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:05.073607] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=20][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:05.073612] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=5][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:05.073616] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:05.073618] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:05.073620] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:05.073621] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:05.073623] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:05.073625] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:05.073627] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:05.073629] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:05.073635] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:05.073637] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:05.073640] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:05.073643] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:05.073645] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:01:05.073647] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:05.073650] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:05.073655] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:05.073660] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:05.073662] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:05.073663] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:05.073668] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:05.073672] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2581][RSAsyncTask2][T1][YB42C0A8002D-0006230057784D8A-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:05.073675] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2581][RSAsyncTask2][T0][YB42C0A8002D-0006230057784D8A-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:01:05.073678] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:05.073681] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:05.073683] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:05.073688] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337665073525, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:05.073691] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:05.073734] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2581][RSAsyncTask2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:01:05.088967] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:05.088976] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.091833] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=8] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:05.119180] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:05.119239] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:05.119239] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=48][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.119246] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:05.119253] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337665119236) [2024-09-26 16:01:05.119259] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337665019051, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:05.119264] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:05.119273] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:05.119276] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:05.119553] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:05.119563] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:05.119566] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:05.119585] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:05.119590] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:05.119592] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:05.119595] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:05.119597] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:05.119599] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:05.119600] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:05.119602] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:05.119603] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:05.119606] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:05.119608] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:05.119614] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:05.119616] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:05.119618] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:05.119620] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:05.119622] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:05.119625] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:05.119629] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:05.119634] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:05.119638] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:05.119640] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:05.119642] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:05.119647] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:05.119650] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:05.119653] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885780-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:05.119657] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:05.119660] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:05.119662] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:05.119664] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337665119497, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:05.119669] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:05.119672] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:05.119696] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:05.119701] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337665119700, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=431, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:05.119708] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:05.119712] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:05.119728] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223394355691, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.119733] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:05.119739] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:05.119744] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:05.119748] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:05.119754] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.119755] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.119757] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337665119715) [2024-09-26 16:01:05.147541] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:221) [2596][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=7] [ASYNC TASK QUEUE](queue_size=0) [2024-09-26 16:01:05.150478] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:05.150487] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.150496] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:05.165095] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337664854674]}) [2024-09-26 16:01:05.172014] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 9290.00) [2024-09-26 16:01:05.172023] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:05.172034] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:05.172038] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:05.172094] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:05.172195] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:05.172823] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=188742, clean_num=31457) [2024-09-26 16:01:05.180752] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:05.180763] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.200330] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:01:05.210888] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:05.210904] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.211751] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:01:05.211766] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=8] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:05.211771] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:01:05.219816] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:05.219826] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337665219811) [2024-09-26 16:01:05.219833] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337665119262, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:05.219853] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223394456307, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.219859] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:05.219867] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:05.219872] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:05.219874] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:05.219880] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.219882] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.219884] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337665219837) [2024-09-26 16:01:05.241339] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14] success to notify_need_writing_throttling True [2024-09-26 16:01:05.241349] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.261190] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:05.261201] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:05.261206] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:05.261210] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:05.261214] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:05.261216] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:05.261220] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:05.261222] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:05.261224] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:05.261226] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:05.261228] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:05.261230] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.261260] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858A9-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:05.265606] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337664854674]}) [2024-09-26 16:01:05.265659] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D9C6-0-0] [lt=6][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337665265597}) [2024-09-26 16:01:05.271691] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:05.271710] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.301864] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:05.301873] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.319955] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:05.319966] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337665319951) [2024-09-26 16:01:05.319982] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337665219836, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:05.319999] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223394555923, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.320005] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:05.320012] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:05.320016] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:05.320019] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:05.320025] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.320027] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.320029] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337665319986) [2024-09-26 16:01:05.321370] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666251596, now=[mts=1727337664321851]) [2024-09-26 16:01:05.321389] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=18][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666251596, now=[mts=1727337664321851], now0=[mts=1727337664321851], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:05.321398] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269166d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337664321226, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:05.321425] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=26][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269166d80}) [2024-09-26 16:01:05.321433] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269166d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337664321226, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:05.321441] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269166d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337664321226, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337664321823, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269166d80}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:05.321471] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=29][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:05.321477] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:05.321480] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:05.321483] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:05.321485] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=1][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:05.321489] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:05.321493] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=1000103, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337666321596) [2024-09-26 16:01:05.321497] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:05.321504] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:05.321506] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:05.321508] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:05.321513] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:05.321546] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:01:05.321549] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:05.332153] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:05.332160] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.355019] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.355030] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337665355017], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:05.355037] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337665355017], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:05.355040] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337665355017]}) [2024-09-26 16:01:05.355045] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:05.355048] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:05.362419] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:05.362426] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.366331] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337665355017]}) [2024-09-26 16:01:05.373287] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:05.373387] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:05.374011] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=220199, clean_num=31457) [2024-09-26 16:01:05.377057] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337664377002]) [2024-09-26 16:01:05.377067] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337664377002], now0=[mts=1727337664377002], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:05.377075] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337664376901, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:05.377095] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=19][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:01:05.377105] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337664376901, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:05.377116] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337664376901, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:05.377138] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=21][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:05.377144] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:05.377147] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:05.377151] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:05.377153] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:01:05.377158] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=27, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:05.377163] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] will sleep(sleep_us=27000, remain_us=1617169, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=27, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:01:05.391692] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=26] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:05.392772] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:05.392781] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.393637] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=8] current monitor number(seq_=-1) [2024-09-26 16:01:05.400523] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:01:05.404392] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:27, local_retry_times:27, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:05.404405] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:05.404409] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:05.404411] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:05.404417] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:05.404479] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:01:05.404485] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:05.416595] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13] ====== tenant freeze timer task ====== [2024-09-26 16:01:05.416605] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:01:05.416636] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337660633362, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337660633318, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337660633383, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:01:05.416658] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:01:05.416723] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E5-0-0] [lt=5] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:01:05.416729] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E5-0-0] [lt=4] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:01:05.416733] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E5-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:01:05.416736] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E5-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:01:05.416740] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E5-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:01:05.416742] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E5-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:01:05.416745] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E5-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:01:05.416753] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E5-0-0] [lt=8] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:01:05.416755] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E5-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:05.416757] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E5-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:05.416760] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E5-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:05.416792] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851E5-0-0] [lt=16] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:01:05.420064] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:05.420078] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337665420062) [2024-09-26 16:01:05.420083] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337665319986, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:05.420100] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223394656526, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.420104] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:05.420111] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:05.420116] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:05.420118] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:05.420123] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.420125] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.420127] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337665420086) [2024-09-26 16:01:05.422824] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:05.422843] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.439646] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D43EE-0-0] [lt=11][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:05.453280] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:05.453298] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.466934] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337665355017]}) [2024-09-26 16:01:05.466990] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.466995] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.467000] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.467531] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.467537] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.467540] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.467544] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.467546] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.467547] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.467549] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=48] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.467551] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.467552] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.468851] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.468854] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.468856] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.468859] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.468860] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.468862] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.468864] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.468865] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.468867] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.469865] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.469872] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.469876] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.469880] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.469882] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.469884] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.469888] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.469890] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.469892] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.470806] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.470812] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.470814] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.470818] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.470820] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.470821] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.470824] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.470825] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.470827] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.471577] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.471582] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.471585] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.471588] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.471589] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.471591] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.471594] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.471595] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.471597] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:05.472768] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.472779] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.472783] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.473779] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.473786] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.473789] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.474543] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.474551] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.474556] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.475696] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.475709] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.475713] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.476678] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.476684] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.476701] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.477661] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.477668] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.477683] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.478624] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.478631] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.478634] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.479370] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.479378] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.479382] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.479545] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO [ratelimit] time: 1727337665479544, bytes: 14724601, bw: 0.004032 MB/s, add_ts: 1006932, add_bytes: 4257 [2024-09-26 16:01:05.480697] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.480706] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.480709] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.481397] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.481406] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.481410] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.482528] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.482540] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.482545] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.483163] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.483172] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.483176] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.483510] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:05.483517] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.484474] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.484484] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.484488] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.485461] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.485465] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.485468] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.486436] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.486444] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.486463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.488468] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.488475] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.488478] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.489362] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.489370] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.489374] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.489920] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.489931] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.489935] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.491495] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.491504] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.491508] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.492305] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.492317] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.492323] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.493272] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.493278] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.493281] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.494245] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.494251] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.494266] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.495257] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.495262] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.495265] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.496211] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.496219] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.496222] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.497204] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.497211] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.497214] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.497721] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.497727] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.497770] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.499143] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.499155] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.499158] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.500107] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.500113] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.500129] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:05.513780] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:05.513791] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.520294] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:05.520332] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337665520291) [2024-09-26 16:01:05.520337] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337665420086, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:05.520369] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=223394755147, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.520375] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:05.520382] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:05.520389] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:05.520392] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:05.520399] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.520401] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.520403] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337665520353) [2024-09-26 16:01:05.522298] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593B-0-0] [lt=31][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:05.522307] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593B-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:05.522333] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593B-0-0] [lt=24][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:05.522338] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593B-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:05.544460] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:05.544472] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.548987] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=1, size_used=6663, mem_used=12478464) [2024-09-26 16:01:05.566513] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=8] gc stale ls task succ [2024-09-26 16:01:05.567375] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337665355017]}) [2024-09-26 16:01:05.574469] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:05.574531] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:05.574538] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.574590] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=51, refresh_score_time=15, wash_time=2) [2024-09-26 16:01:05.575216] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=251656, clean_num=31457) [2024-09-26 16:01:05.590966] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [2619][DumpLSLoc][T0][YB42C0A8002D-0006230058384C7C-0-0] [lt=14] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1727337660936971, replica_locations:[{server:"192.168.0.45:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:374}]}]) [2024-09-26 16:01:05.594873] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=15] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:01:05.600696] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:01:05.600741] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.601686] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.601720] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.601730] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.602392] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [2635][LocalityReload][T0][Y0-0000000000000000-0-0] [lt=7] runTimer to refresh locality_info(ret=0) [2024-09-26 16:01:05.602782] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7C-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:05.602795] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7C-0-0] [lt=11][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.602825] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2634][LocltyRefTask][T1][YB42C0A8002D-0006230058284C7C-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:05.602835] INFO [SHARE] load_region (ob_locality_table_operator.cpp:373) [2634][LocltyRefTask][T0][Y0-0000000000000000-0-0] [lt=0] load region(ret=-5019, locality_info={version:0, local_region:"", local_zone:"", local_idc:"", local_zone_type:3, local_zone_status:3, locality_region_array:[], locality_zone_array:[]}) [2024-09-26 16:01:05.604668] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:05.604676] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.613459] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:05.620320] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:05.620333] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:05.620337] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:05.620664] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885781-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:05.620675] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885781-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.620708] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885781-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:05.620734] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337665620734, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=405, wlock_time=10, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:05.620744] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:05.620760] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223394855740, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.620765] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:05.620771] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:05.620777] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:05.620779] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:05.620785] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.620787] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.620790] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337665620748) [2024-09-26 16:01:05.634908] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:05.634932] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] ====== traversal_flush timer task ====== [2024-09-26 16:01:05.634917] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.634967] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5] add dag success(dag=0x7f9264d76030, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6163825430799493735, dag_cnt=1, dag_type_cnts=1) [2024-09-26 16:01:05.634977] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] success to create and add dag(ret=0, dag=0x7f9264d76030) [2024-09-26 16:01:05.634980] INFO [STORAGE] flush (ob_tx_data_memtable.cpp:917) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] [TX DATA MERGE]schedule flush tx data memtable task done(ret=0, ret="OB_SUCCESS", get_ls_id()={id:1}, this=0x7f92a2294080, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:4, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337665634980, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:01:05.634995] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337665634989, task_id:YB42C0A8002D-0006230056D8C72A-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.635012] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=15] schedule one task(task=0x7f9264d763f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.635016] INFO [MDS] flush (mds_table_mgr.cpp:172) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=19] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:01:05.635042] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.635047] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.635050] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726157244792838000) [2024-09-26 16:01:05.635054] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=20] add dag success(dag=0x7f9264d76560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6659188001345271846, dag_cnt=2, dag_type_cnts=1) [2024-09-26 16:01:05.635057] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d76560) [2024-09-26 16:01:05.635060] INFO [STORAGE] ready_for_flush (ob_tx_data_memtable.cpp:865) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=2] memtable is frozen yet.(this=0x7f92a2294080) [2024-09-26 16:01:05.635061] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, create_flush_dag_time:1727337665635061, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452867, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200240}, checkpoint_unit_ptr:0x7f92a1a70850}, is_freeze_clock:true}) [2024-09-26 16:01:05.635068] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337665635046, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635085] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=13] add dag success(dag=0x7f9264d76a90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-9069613569758001279, dag_cnt=3, dag_type_cnts=2) [2024-09-26 16:01:05.635089] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d76a90) [2024-09-26 16:01:05.635091] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, create_flush_dag_time:1727337665635091, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452892, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200489}, checkpoint_unit_ptr:0x7f92a1a70080}, is_freeze_clock:true}) [2024-09-26 16:01:05.635094] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337665635081, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635104] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d76fc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1288986994123435108, dag_cnt=4, dag_type_cnts=3) [2024-09-26 16:01:05.635109] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] success to create and add dag(ret=0, dag=0x7f9264d76fc0) [2024-09-26 16:01:05.635111] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337665635111, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452911, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847222}, checkpoint_unit_ptr:0x7f92a1a71020}, is_freeze_clock:true}) [2024-09-26 16:01:05.635114] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337665635102, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635123] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d774f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5355160120942094702, dag_cnt=5, dag_type_cnts=4) [2024-09-26 16:01:05.635125] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d774f0) [2024-09-26 16:01:05.635127] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337665635127, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452929, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847223}, checkpoint_unit_ptr:0x7f92a1a717f0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635130] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337665635121, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635139] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405042) [2024-09-26 16:01:05.635150] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:01:05.635153] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405070) [2024-09-26 16:01:05.635160] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:01:05.635163] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405087) [2024-09-26 16:01:05.635170] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:01:05.635172] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405104) [2024-09-26 16:01:05.635179] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:01:05.635183] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] add dag success(dag=0x7f9264d77a20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1994541446894784994, dag_cnt=6, dag_type_cnts=5) [2024-09-26 16:01:05.635185] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d77a20) [2024-09-26 16:01:05.635187] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, create_flush_dag_time:1727337665635187, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209556884, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:226}, checkpoint_unit_ptr:0x7f92a1a73f00}, is_freeze_clock:true}) [2024-09-26 16:01:05.635190] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337665635180, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635203] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d77f50, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8791616776033068452, dag_cnt=7, dag_type_cnts=6) [2024-09-26 16:01:05.635205] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d77f50) [2024-09-26 16:01:05.635206] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337665635206, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405141, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847408}, checkpoint_unit_ptr:0x7f92a1a746d0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635209] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337665635197, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635218] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d78480, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6550230011160095889, dag_cnt=8, dag_type_cnts=7) [2024-09-26 16:01:05.635220] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d78480) [2024-09-26 16:01:05.635221] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337665635221, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405158, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847409}, checkpoint_unit_ptr:0x7f92a1a75670}, is_freeze_clock:true}) [2024-09-26 16:01:05.635224] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337665635216, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635233] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d789b0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6809498982591245614, dag_cnt=9, dag_type_cnts=8) [2024-09-26 16:01:05.635235] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d789b0) [2024-09-26 16:01:05.635236] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, create_flush_dag_time:1727337665635236, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405174, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:200004}, checkpoint_unit_ptr:0x7f92a1a74ea0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635239] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337665635231, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635248] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d78ee0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6815944285528637599, dag_cnt=10, dag_type_cnts=9) [2024-09-26 16:01:05.635250] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d78ee0) [2024-09-26 16:01:05.635251] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, create_flush_dag_time:1727337665635251, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209506214, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1}, checkpoint_unit_ptr:0x7f92a1a76610}, is_freeze_clock:true}) [2024-09-26 16:01:05.635254] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337665635246, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635263] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d79410, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-4182959058374342994, dag_cnt=11, dag_type_cnts=10) [2024-09-26 16:01:05.635276] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=13] success to create and add dag(ret=0, dag=0x7f9264d79410) [2024-09-26 16:01:05.635278] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, create_flush_dag_time:1727337665635278, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405204, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:412}, checkpoint_unit_ptr:0x7f92a1a75e40}, is_freeze_clock:true}) [2024-09-26 16:01:05.635282] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337665635261, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635292] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d79940, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1586518267935340, dag_cnt=12, dag_type_cnts=11) [2024-09-26 16:01:05.635294] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d79940) [2024-09-26 16:01:05.635296] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, create_flush_dag_time:1727337665635296, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405220, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:444}, checkpoint_unit_ptr:0x7f92a1a775b0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635299] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337665635289, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635307] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d79e70, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3044367941103776476, dag_cnt=13, dag_type_cnts=12) [2024-09-26 16:01:05.635309] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d79e70) [2024-09-26 16:01:05.635311] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, create_flush_dag_time:1727337665635311, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455964, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:309}, checkpoint_unit_ptr:0x7f92a1a76de0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635314] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337665635305, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635322] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7a3a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1596830851104282047, dag_cnt=14, dag_type_cnts=13) [2024-09-26 16:01:05.635324] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7a3a0) [2024-09-26 16:01:05.635325] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, create_flush_dag_time:1727337665635325, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455986, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:331}, checkpoint_unit_ptr:0x7f92a1a77d80}, is_freeze_clock:true}) [2024-09-26 16:01:05.635328] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337665635320, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635337] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7a8d0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2360195962081676673, dag_cnt=15, dag_type_cnts=14) [2024-09-26 16:01:05.635339] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d7a8d0) [2024-09-26 16:01:05.635340] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, create_flush_dag_time:1727337665635340, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405262, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:458}, checkpoint_unit_ptr:0x7f92a1a7a490}, is_freeze_clock:true}) [2024-09-26 16:01:05.635343] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337665635335, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635352] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7ae00, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5675615614797020460, dag_cnt=16, dag_type_cnts=15) [2024-09-26 16:01:05.635353] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d7ae00) [2024-09-26 16:01:05.635355] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, create_flush_dag_time:1727337665635355, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405279, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:487}, checkpoint_unit_ptr:0x7f92a1a79cc0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635358] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337665635349, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635380] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=18] add dag success(dag=0x7f9264d7b330, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-612435342908088006, dag_cnt=17, dag_type_cnts=16) [2024-09-26 16:01:05.635382] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7b330) [2024-09-26 16:01:05.635384] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, create_flush_dag_time:1727337665635384, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405296, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:455}, checkpoint_unit_ptr:0x7f92a1a794f0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635387] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337665635376, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635392] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337665635388, task_id:YB42C0A8002D-0006230056D8C72B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.635399] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=6] schedule one task(task=0x7f9264d76920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.635405] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337665635404, task_id:YB42C0A8002D-0006230056D8C72C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.635408] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2] schedule one task(task=0x7f9264d76e50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.635413] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337665635413, task_id:YB42C0A8002D-0006230056D8C72D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.635416] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=3] schedule one task(task=0x7f9264d77380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.635419] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665635418, task_id:YB42C0A8002D-0006230056D8C72E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.635421] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=2] schedule one task(task=0x7f9264d778b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.635424] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665635424, task_id:YB42C0A8002D-0006230056D8C72F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.635426] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=2] schedule one task(task=0x7f9264d77de0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.635449] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.635449] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.635453] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.635454] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.635455] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:01:05.635457] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.635457] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.635461] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.635461] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.635462] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.635463] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:01:05.635468] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.635467] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:01:05.635465] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847222}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:01:05.635472] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:01:05.635476] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=6] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.635478] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241777443612000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.635497] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.635515] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.635518] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.635519] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:01:05.635519] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.635522] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.635523] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.635524] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:01:05.635527] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.635536] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7b860, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7868818205990873505, dag_cnt=18, dag_type_cnts=17) [2024-09-26 16:01:05.635539] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264d7b860) [2024-09-26 16:01:05.635541] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, create_flush_dag_time:1727337665635541, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405313, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:492}, checkpoint_unit_ptr:0x7f92a1a78550}, is_freeze_clock:true}) [2024-09-26 16:01:05.635545] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337665635394, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635559] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264d7bd90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7071731214611448811, dag_cnt=19, dag_type_cnts=18) [2024-09-26 16:01:05.635561] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7bd90) [2024-09-26 16:01:05.635563] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, create_flush_dag_time:1727337665635563, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405347, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:501}, checkpoint_unit_ptr:0x7f92a1a78d20}, is_freeze_clock:true}) [2024-09-26 16:01:05.635566] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337665635553, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635575] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7c2c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8731750006000903454, dag_cnt=20, dag_type_cnts=19) [2024-09-26 16:01:05.635577] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7c2c0) [2024-09-26 16:01:05.635578] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, create_flush_dag_time:1727337665635578, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405365, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:472}, checkpoint_unit_ptr:0x7f92a1a7ac60}, is_freeze_clock:true}) [2024-09-26 16:01:05.635581] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337665635573, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635589] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7c7f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6766388540859538791, dag_cnt=21, dag_type_cnts=20) [2024-09-26 16:01:05.635591] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7c7f0) [2024-09-26 16:01:05.635592] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, create_flush_dag_time:1727337665635592, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405384, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:342}, checkpoint_unit_ptr:0x7f92a1a7cba0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635595] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337665635587, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635604] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7cd20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2986199386776543176, dag_cnt=22, dag_type_cnts=21) [2024-09-26 16:01:05.635605] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d7cd20) [2024-09-26 16:01:05.635607] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337665635607, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405418, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:140}, checkpoint_unit_ptr:0x7f92a1a7b430}, is_freeze_clock:true}) [2024-09-26 16:01:05.635610] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337665635602, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635622] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264d7d250, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6947063304478012270, dag_cnt=23, dag_type_cnts=22) [2024-09-26 16:01:05.635624] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7d250) [2024-09-26 16:01:05.635626] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337665635625, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405436, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101009}, checkpoint_unit_ptr:0x7f92a1a7bc00}, is_freeze_clock:true}) [2024-09-26 16:01:05.635629] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337665635619, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635638] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7d780, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5843378789243662202, dag_cnt=24, dag_type_cnts=23) [2024-09-26 16:01:05.635640] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d7d780) [2024-09-26 16:01:05.635641] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337665635641, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405453, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101010}, checkpoint_unit_ptr:0x7f92a1a7c3d0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635644] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337665635635, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635652] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7dcb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1841823684513746176, dag_cnt=25, dag_type_cnts=24) [2024-09-26 16:01:05.635654] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7dcb0) [2024-09-26 16:01:05.635656] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, create_flush_dag_time:1727337665635656, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405470, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:323}, checkpoint_unit_ptr:0x7f92a1a7d370}, is_freeze_clock:true}) [2024-09-26 16:01:05.635659] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337665635650, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635667] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7e1e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-2678407261265885087, dag_cnt=26, dag_type_cnts=25) [2024-09-26 16:01:05.635669] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7e1e0) [2024-09-26 16:01:05.635670] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, create_flush_dag_time:1727337665635670, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405487, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:377}, checkpoint_unit_ptr:0x7f92a1a7e310}, is_freeze_clock:true}) [2024-09-26 16:01:05.635673] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337665635665, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635681] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7e710, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7175089551615529855, dag_cnt=27, dag_type_cnts=26) [2024-09-26 16:01:05.635683] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7e710) [2024-09-26 16:01:05.635685] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, create_flush_dag_time:1727337665635685, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405503, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:297}, checkpoint_unit_ptr:0x7f92a1a7eae0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635688] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337665635679, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635696] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7ec40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3730526293242960203, dag_cnt=28, dag_type_cnts=27) [2024-09-26 16:01:05.635698] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7ec40) [2024-09-26 16:01:05.635699] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337665635699, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405519, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101065}, checkpoint_unit_ptr:0x7f92a1a7db40}, is_freeze_clock:true}) [2024-09-26 16:01:05.635702] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337665635694, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635711] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d7f170, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=4663792628280492528, dag_cnt=29, dag_type_cnts=28) [2024-09-26 16:01:05.635712] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d7f170) [2024-09-26 16:01:05.635714] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337665635714, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405536, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:333}, checkpoint_unit_ptr:0x7f92a3bc4850}, is_freeze_clock:true}) [2024-09-26 16:01:05.635717] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337665635709, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635725] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7f6a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8399557287176188414, dag_cnt=30, dag_type_cnts=29) [2024-09-26 16:01:05.635727] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7f6a0) [2024-09-26 16:01:05.635729] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337665635729, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405552, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101066}, checkpoint_unit_ptr:0x7f92a3bc5020}, is_freeze_clock:true}) [2024-09-26 16:01:05.635732] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337665635723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635740] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d7fbd0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8068938909778324349, dag_cnt=31, dag_type_cnts=30) [2024-09-26 16:01:05.635742] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d7fbd0) [2024-09-26 16:01:05.635743] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, create_flush_dag_time:1727337665635743, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405569, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:334}, checkpoint_unit_ptr:0x7f92a3bc57f0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635746] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337665635738, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635755] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d80100, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7665372330263152188, dag_cnt=32, dag_type_cnts=31) [2024-09-26 16:01:05.635756] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d80100) [2024-09-26 16:01:05.635758] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, create_flush_dag_time:1727337665635758, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405586, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:451}, checkpoint_unit_ptr:0x7f92a3bc4080}, is_freeze_clock:true}) [2024-09-26 16:01:05.635761] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337665635753, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635769] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d80630, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6759548225246904317, dag_cnt=33, dag_type_cnts=32) [2024-09-26 16:01:05.635771] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d80630) [2024-09-26 16:01:05.635773] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, create_flush_dag_time:1727337665635772, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405602, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:378}, checkpoint_unit_ptr:0x7f92a1a7f2b0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635775] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337665635767, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635784] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d80b60, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8944614441376604790, dag_cnt=34, dag_type_cnts=33) [2024-09-26 16:01:05.635786] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d80b60) [2024-09-26 16:01:05.635787] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, create_flush_dag_time:1727337665635787, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209456003, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:121}, checkpoint_unit_ptr:0x7f92a3bc5fc0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635790] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337665635782, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635799] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d81090, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=507863347806783454, dag_cnt=35, dag_type_cnts=34) [2024-09-26 16:01:05.635800] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d81090) [2024-09-26 16:01:05.635809] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, create_flush_dag_time:1727337665635809, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405633, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:215}, checkpoint_unit_ptr:0x7f92a3bc86d0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635812] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337665635796, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635822] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d815c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1604282355669434429, dag_cnt=36, dag_type_cnts=35) [2024-09-26 16:01:05.635824] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d815c0) [2024-09-26 16:01:05.635826] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, create_flush_dag_time:1727337665635826, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405649, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:456}, checkpoint_unit_ptr:0x7f92a3bc6790}, is_freeze_clock:true}) [2024-09-26 16:01:05.635829] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337665635819, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635838] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d81af0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3553649416163308074, dag_cnt=37, dag_type_cnts=36) [2024-09-26 16:01:05.635840] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d81af0) [2024-09-26 16:01:05.635841] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337665635841, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405666, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:154}, checkpoint_unit_ptr:0x7f92a3bc6f60}, is_freeze_clock:true}) [2024-09-26 16:01:05.635844] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337665635836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635853] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264d82020, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8097406237500329300, dag_cnt=38, dag_type_cnts=37) [2024-09-26 16:01:05.635854] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d82020) [2024-09-26 16:01:05.635856] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337665635856, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405682, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101015}, checkpoint_unit_ptr:0x7f92a3bc7730}, is_freeze_clock:true}) [2024-09-26 16:01:05.635859] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337665635851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635868] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d82550, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3976350754940852513, dag_cnt=39, dag_type_cnts=38) [2024-09-26 16:01:05.635869] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d82550) [2024-09-26 16:01:05.635871] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337665635871, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405699, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101016}, checkpoint_unit_ptr:0x7f92a3bc7f00}, is_freeze_clock:true}) [2024-09-26 16:01:05.635874] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337665635865, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635883] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d82a80, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8720304653345806796, dag_cnt=40, dag_type_cnts=39) [2024-09-26 16:01:05.635884] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d82a80) [2024-09-26 16:01:05.635886] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, create_flush_dag_time:1727337665635886, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405716, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101063}, checkpoint_unit_ptr:0x7f92a3bca610}, is_freeze_clock:true}) [2024-09-26 16:01:05.635889] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337665635881, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635898] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d82fb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1830562711513853292, dag_cnt=41, dag_type_cnts=40) [2024-09-26 16:01:05.635899] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d82fb0) [2024-09-26 16:01:05.635901] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, create_flush_dag_time:1727337665635901, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405733, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:452}, checkpoint_unit_ptr:0x7f92a3bc9e40}, is_freeze_clock:true}) [2024-09-26 16:01:05.635904] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337665635895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635912] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d834e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1536604874905918360, dag_cnt=42, dag_type_cnts=41) [2024-09-26 16:01:05.635914] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d834e0) [2024-09-26 16:01:05.635915] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337665635915, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405750, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:328}, checkpoint_unit_ptr:0x7f92a3bc8ea0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635918] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337665635910, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635927] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d83a10, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6933628521298738662, dag_cnt=43, dag_type_cnts=42) [2024-09-26 16:01:05.635928] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d83a10) [2024-09-26 16:01:05.635930] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337665635930, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405767, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:330}, checkpoint_unit_ptr:0x7f92a3bc9670}, is_freeze_clock:true}) [2024-09-26 16:01:05.635933] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337665635924, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635941] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d83f40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1056199536334423564, dag_cnt=44, dag_type_cnts=43) [2024-09-26 16:01:05.635943] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d83f40) [2024-09-26 16:01:05.635944] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337665635944, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405784, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:329}, checkpoint_unit_ptr:0x7f92a3bcb5b0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635947] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337665635939, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635957] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d84470, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7405463607918801432, dag_cnt=45, dag_type_cnts=44) [2024-09-26 16:01:05.635959] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264d84470) [2024-09-26 16:01:05.635961] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, create_flush_dag_time:1727337665635961, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405801, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:332}, checkpoint_unit_ptr:0x7f92a3bcade0}, is_freeze_clock:true}) [2024-09-26 16:01:05.635964] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337665635953, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635973] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264d849a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6699723950958743903, dag_cnt=46, dag_type_cnts=45) [2024-09-26 16:01:05.635974] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264d849a0) [2024-09-26 16:01:05.635976] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, create_flush_dag_time:1727337665635976, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405818, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101067}, checkpoint_unit_ptr:0x7f92a3bcbd80}, is_freeze_clock:true}) [2024-09-26 16:01:05.635979] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337665635970, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.635985] INFO [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:655) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] traversal_flush successfully(ls_->get_ls_id()={id:1}, flush_tasks=[{table:0x7f92a1a70850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a70080, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71020, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a717f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71fc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72790, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72f60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73730, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73f00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a746d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75670, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a74ea0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76610, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75e40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a775b0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76de0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a77d80, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7a490, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a79cc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a794f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78550, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78d20, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7ac60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7cba0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7b430, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7bc00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7c3d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7d370, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7e310, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7eae0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7db40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a3bc4850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}]) [2024-09-26 16:01:05.636044] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=56] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-09-26 16:01:05.646639] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1, storage_schema:{this:0x7f9263a432b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.646657] INFO [STORAGE] prepare_tx_data_list (ob_tx_data_memtable.cpp:266) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=17] construct list more than once. skip this time(this=0x7f92a2294080) [2024-09-26 16:01:05.648716] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:01:05.648729] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=7] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.648796] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=22] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337665635180, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.648814] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=17] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752755184, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828752755184, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.648834] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=12] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.648838] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.648847] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:226}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752755184, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.648873] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=18] succeed to build merge ctx(tablet_id={id:226}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752755184, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:226}, table_count:1, [{i:0, table_key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}, merge_scn:{val:1726241777443612000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665635443, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=13.40ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.648908] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=26] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72F-0-0, dag_ret:0, dag_status:2, start_time:1727337665635426, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3480, output_block_cnt_arr:0x7f9287db3488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:56623104, latest_update_ts:1727337665635443, estimated_finish_time:1727337682648935}) [2024-09-26 16:01:05.648916] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d77de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72F-0-0, dag_ret:0, dag_status:2, start_time:1727337665635426, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:01:05.648922] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=6] task finish process(ret=0, start_time=1727337665635439, end_time=1727337665648922, runtime=13483, *this={this:0x7f9264d77de0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72F-0-0, dag_ret:0, dag_status:2, start_time:1727337665635426, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:01:05.651234] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.651251] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=17] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337665635102, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:01:05.651259] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.651353] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:01:05.651357] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.651364] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847222}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:01:05.651388] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:1152921504606847222}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847222}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665635444, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=15.92ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.651537] INFO [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:329) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=3] pre-process commit versions row(get_ls_id()={id:1}, do_recycle_=true, current_time=1727337665651535, prev_recycle_time=0) [2024-09-26 16:01:05.651691] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=24] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:01:05.652354] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=4] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=8366, total_rows=58562) [2024-09-26 16:01:05.652821] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72D-0-0, dag_ret:0, dag_status:2, start_time:1727337665635416, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505470, output_block_cnt_arr:0x7f9266505478, concurrent_cnt:1, estimate_row_cnt:329211, estimate_occupy_size:150994944, latest_update_ts:1727337665635444, estimated_finish_time:1727337682652892}) [2024-09-26 16:01:05.652837] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=15] succeed to init merge ctx(task={this:0x7f9264d77380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72D-0-0, dag_ret:0, dag_status:2, start_time:1727337665635416, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.652844] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=6] task finish process(ret=0, start_time=1727337665635442, end_time=1727337665652844, runtime=17402, *this={this:0x7f9264d77380, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72D-0-0, dag_ret:0, dag_status:2, start_time:1727337665635416, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.652899] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.652925] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=25] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337665635081, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=17, max_schema_version_on_memtable=1725120002010696) [2024-09-26 16:01:05.652933] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=17, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.653034] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1887, total_rows=92463) [2024-09-26 16:01:05.653038] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.653045] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200489}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:19, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.653039] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.653069] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=29] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337665635121, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:01:05.653066] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:200489}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200489}, table_count:2, [{i:0, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665635512, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=17.53ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.653078] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.653179] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:01:05.653183] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.653190] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847223}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:01:05.653213] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:1152921504606847223}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847223}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665635451, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=17.73ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.653374] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72C-0-0, dag_ret:0, dag_status:2, start_time:1727337665635407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a45460, output_block_cnt_arr:0x7f9263a45468, concurrent_cnt:1, estimate_row_cnt:254187, estimate_occupy_size:138412032, latest_update_ts:1727337665635512, estimated_finish_time:1727337682653440}) [2024-09-26 16:01:05.653399] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=24] succeed to init merge ctx(task={this:0x7f9264d76e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72C-0-0, dag_ret:0, dag_status:2, start_time:1727337665635407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.653405] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=6] task finish process(ret=0, start_time=1727337665635510, end_time=1727337665653405, runtime=17895, *this={this:0x7f9264d76e50, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72C-0-0, dag_ret:0, dag_status:2, start_time:1727337665635407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.653513] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=23] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:01:05.653791] INFO [STORAGE] get_ls_min_end_scn (ob_ls_tablet_service.cpp:5618) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=6] get ls min end scn finish(ls_checkpoint={val:1726163586617842000, v:0}) [2024-09-26 16:01:05.653802] WDIAG [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:761) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=6][errcode=-4023] failed to get snapshot version(ret=-4023, MTL_ID()=1) [2024-09-26 16:01:05.653806] INFO [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:773) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=4] get tx data recycle scn finish.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, recycle_scn={val:1726076389100953000, v:0}, min_end_scn_from_old_tablets={val:4611686018427387903, v:0}, min_end_scn_from_latest_tablets={val:1726076389100953000, v:0}) [2024-09-26 16:01:05.653812] WDIAG [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:336) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=5][errcode=-4023] get recycle ts failed.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:01:05.653817] WDIAG [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:236) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=2][errcode=-4023] process commit version row failed.(ret=-4023, ret="OB_EAGAIN", this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337665634980, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:01:05.653829] INFO [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:254) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=11] [TX DATA MERGE]pre process for merge done(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}, this=0x7f92a2294080, tg=time guard have no click events for optimization, fake_tx_data_guard={tx_data:{tx_id:{txid:9223372036854775807}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337665634980, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:01:05.653841] WDIAG [STORAGE.COMPACTION] pre_process_tx_data_table_merge_ (ob_tx_table_merge_task.cpp:117) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=11][errcode=-4023] do pre process for tx data table merge failed.(ret=-4023, ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, table={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337665634980, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:01:05.653849] WDIAG [STORAGE.COMPACTION] inner_init_ctx (ob_tx_table_merge_task.cpp:157) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=8][errcode=-4023] pre process tx data table for merge failed.(ret=-4023, ret="OB_EAGAIN", ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.653853] WDIAG [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1061) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=2][errcode=-4023] fail to inner init ctx(ret=-4023, tablet_id={id:49402}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, create_snapshot_version:0, is_full_merge:false, merge_level:1, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:4, range_array:[], concurrent_cnt:0, is_inited:false}, schema_ctx:{schema_version:1, storage_schema:{this:0x7f9263a432b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:49402}, table_count:1, [{i:0, table_key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref:4}]}, schedule_major:false, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}, merge_scn:{val:4611686018427387903, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:NULL, compaction_filter:NULL, time_guard:total=18.81ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}) [2024-09-26 16:01:05.653871] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:975) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=18][errcode=-4023] failed to build merge ctx(ret=-4023, ctx->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.653874] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:994) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=3][errcode=-4023] sstable merge finish(ret=-4023, ctx=0x7f9263a42060, task={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C72A-0-0, dag_ret:0, dag_status:2, start_time:1727337665635011, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:01:05.653882] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=7][errcode=-4023] failed to process task(ret=-4023) [2024-09-26 16:01:05.653884] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=2] task finish process(ret=-4023, start_time=1727337665635034, end_time=1727337665653884, runtime=18850, *this={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C72A-0-0, dag_ret:0, dag_status:2, start_time:1727337665635011, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:01:05.653890] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=6][errcode=-4023] failed to do work(ret=-4023, *task_={this:0x7f9264d763f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C72A-0-0, dag_ret:0, dag_status:2, start_time:1727337665635011, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.653898] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=5] dag finished(dag_ret=-4023, runtime=18887, dag_cnt=45, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264d76030, dag={ObIDag:{this:0x7f9264d76030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C72A-0-0, dag_ret:-4023, dag_status:5, start_time:1727337665635011, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}) [2024-09-26 16:01:05.653911] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2873][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C72A-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337665634989, task_id:YB42C0A8002D-0006230056D8C72A-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.653955] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726158220122116000) [2024-09-26 16:01:05.653969] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.654203] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=5] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=7999, total_rows=55993) [2024-09-26 16:01:05.654612] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72E-0-0, dag_ret:0, dag_status:2, start_time:1727337665635421, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab470, output_block_cnt_arr:0x7f92684ab478, concurrent_cnt:1, estimate_row_cnt:332829, estimate_occupy_size:150994944, latest_update_ts:1727337665635451, estimated_finish_time:1727337682654683}) [2024-09-26 16:01:05.654624] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=12] succeed to init merge ctx(task={this:0x7f9264d778b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72E-0-0, dag_ret:0, dag_status:2, start_time:1727337665635421, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.654642] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=18] task finish process(ret=0, start_time=1727337665635449, end_time=1727337665654642, runtime=19193, *this={this:0x7f9264d778b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72E-0-0, dag_ret:0, dag_status:2, start_time:1727337665635421, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.654736] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.654743] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=5] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.654745] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=1] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.654748] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=2] schedule one task(task=0x7f9264d84ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.654756] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337665654752, task_id:YB42C0A8002D-0006230056D8C730-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.654760] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=4] schedule one task(task=0x7f9264d78310, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.654837] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5e060, count:4, allocator:0x7f92a31f67d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264d5e350, local_datums:0x7f9264d5e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:05.654837] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5c060, count:3, allocator:0x7f92636927d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264d5c320, local_datums:0x7f9264d5c320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:01:05.654845] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=226, range=0x7f92aaf52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fc060, count:5, allocator:0x7f9269ffe7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fc360, local_datums:0x7f92665fc360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:01:05.654861] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=21] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d62060, count:4, allocator:0x7f92a31f7558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264d62350, local_datums:0x7f9264d62350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:05.654865] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=22] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264b22060, count:3, allocator:0x7f9263693558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264b22320, local_datums:0x7f9264b22320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:01:05.655026] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.655094] ERROR issue_dba_error (ob_log.cpp:1875) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.655098] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=7][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.655097] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.655104] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.655106] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.655108] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264dd0070, hex: 9F094D0300000000, int: 55380383},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264dd00a8, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264dd00e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3044, column_count:19, rowkey_column_count:3, has_column_checksum:0, row_count:83, row_store_type:0, opt:5, var_column_count:0, row_offset:16109, original_length:16381, max_merged_trans_version:1726163995623997000, data_length:16381, data_zlength:7759, data_checksum:3416849541, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9264dd2030, buf_size:7759, data_size:16381, row_count:83, column_count:19, max_merged_trans_version:1726163995623997000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3537734793, row_count_delta:83, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16381}) [2024-09-26 16:01:05.655123] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=15][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:05.655125] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:05.655127] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:05.655129] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:05.655147] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.655151] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.655159] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.655161] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.655163] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9264dce140, hex: 3230323430393131303133333131, cstr: 20240911013311},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9264dce14e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264dce0e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264dce118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-15038, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7227, data_checksum:1452123993, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f926585e030, buf_size:7227, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1818264584, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:01:05.655175] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=12][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:05.655177] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:05.655178] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:05.655180] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:05.655130] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264b22320, local_datums:0x7f9264b22320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b223a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264b224f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b225d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264b226e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264b22320, local_datums:0x7f9264b22320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b223a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264b224f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b225d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264b226e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f9263692e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263692e98, limit_param:null, stmt_allocator:0x7f9263693558, allocator:0x7f9263693558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:-1731361377836159744, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264b22320, local_datums:0x7f9264b22320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b223a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264b224f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b225d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264b226e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337665635081, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}) [2024-09-26 16:01:05.655201] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=71][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:05.655182] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264d5e350, local_datums:0x7f9264d5e350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5e3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5e408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264d5e350, local_datums:0x7f9264d5e350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5e3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5e408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92a31f6118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31f6118, limit_param:null, stmt_allocator:0x7f92a31f67d8, allocator:0x7f92a31f67d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264d5e350, local_datums:0x7f9264d5e350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5e3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5e408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:01:05.655236] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=54][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:05.655203] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264b22320, local_datums:0x7f9264b22320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b223a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264b224f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b225d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264b226e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f9263692e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263692e98, limit_param:null, stmt_allocator:0x7f9263693558, allocator:0x7f9263693558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:-1731361377836159744, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f9264b22320, local_datums:0x7f9264b22320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b223a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f9264b224f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b22560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264b225d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f9264b226e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337665635081, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}]) [2024-09-26 16:01:05.655242] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=39][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:05.655244] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.655257] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=10] partition merge iter row count(i=1, row_count=83, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, table={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337665635081, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.655266] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:05.655238] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264d5e350, local_datums:0x7f9264d5e350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5e3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5e408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92a31f6118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31f6118, limit_param:null, stmt_allocator:0x7f92a31f67d8, allocator:0x7f92a31f67d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264d5e350, local_datums:0x7f9264d5e350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5e3d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d5e408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:01:05.655269] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=31][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:05.655271] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=1] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.655278] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:05.655281] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:05.655283] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=9] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337665635121, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.655283] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337665654773, end_time=1727337665655283, runtime=510, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72C-0-0, dag_ret:0, dag_status:2, start_time:1727337665635407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.655292] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:05.655290] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72C-0-0, dag_ret:0, dag_status:2, start_time:1727337665635407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.655298] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=19891, dag_cnt=44, dag_cnts_[dag.get_type()]=44, &dag=0x7f9264d76a90, dag={ObIDag:{this:0x7f9264d76a90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665635407, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.655303] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:05.655306] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:05.655307] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72C-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337665635404, task_id:YB42C0A8002D-0006230056D8C72C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.655308] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337665654756, end_time=1727337665655308, runtime=552, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72E-0-0, dag_ret:0, dag_status:2, start_time:1727337665635421, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.655315] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72E-0-0, dag_ret:0, dag_status:2, start_time:1727337665635421, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.655359] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.655365] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.655368] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.655374] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.655394] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=19973, dag_cnt=43, dag_cnts_[dag.get_type()]=43, &dag=0x7f9264d774f0, dag={ObIDag:{this:0x7f9264d774f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665635421, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.655412] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72E-0-0] [lt=15] succeed to del sys task(removed_task={start_time:1727337665635418, task_id:YB42C0A8002D-0006230056D8C72E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.655412] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:4, allocator:0x7f92a33567d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684b4350, local_datums:0x7f92684b4350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:05.655430] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=15] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b2060, count:4, allocator:0x7f92a3357558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684b2350, local_datums:0x7f92684b2350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:05.655683] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=7][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.655718] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.655721] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.655728] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.655730] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.655731] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f926859a140, hex: 3230323430393131303133333438, cstr: 20240911013348},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f926859a14e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f926859a0e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f926859a118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3080, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7221, data_checksum:4284205003, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9268560030, buf_size:7221, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1004563202, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:01:05.655742] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:05.655743] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:05.655745] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:05.655748] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=3][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:05.655749] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684b4350, local_datums:0x7f92684b4350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684b4350, local_datums:0x7f92684b4350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92a3356118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a3356118, limit_param:null, stmt_allocator:0x7f92a33567d8, allocator:0x7f92a33567d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684b4350, local_datums:0x7f92684b4350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:01:05.655810] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=60][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:05.655812] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684b4350, local_datums:0x7f92684b4350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92a3356118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a3356118, limit_param:null, stmt_allocator:0x7f92a33567d8, allocator:0x7f92a33567d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684b4350, local_datums:0x7f92684b4350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:01:05.655842] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=30][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:05.655844] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=1] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.655854] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=7] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337665635102, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.655862] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:05.655872] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:05.655875] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:05.655877] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337665655354, end_time=1727337665655876, runtime=522, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72D-0-0, dag_ret:0, dag_status:2, start_time:1727337665635416, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.655883] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72D-0-0, dag_ret:0, dag_status:2, start_time:1727337665635416, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.655890] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=20474, dag_cnt=42, dag_cnts_[dag.get_type()]=42, &dag=0x7f9264d76fc0, dag={ObIDag:{this:0x7f9264d76fc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665635416, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.655898] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72D-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337665635413, task_id:YB42C0A8002D-0006230056D8C72D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.655972] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665655970, task_id:YB42C0A8002D-0006230056D8C731-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.655975] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=3] schedule one task(task=0x7f9264d78840, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.655981] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337665655980, task_id:YB42C0A8002D-0006230056D8C732-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.655983] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=2] schedule one task(task=0x7f9264d78d70, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.655989] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665655988, task_id:YB42C0A8002D-0006230056D8C733-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.655991] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=2] schedule one task(task=0x7f9264d792a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.656018] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.656022] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.656024] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.656024] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.656028] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.656028] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.656030] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.656031] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.656035] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.656040] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.656060] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.656062] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.656064] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.656067] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.656069] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.656073] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241621541056000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.661364] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=14][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.661384] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:01:05.661398] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=8] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.661411] ERROR issue_dba_error (ob_log.cpp:1875) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=9][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.661414] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.661421] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.661423] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.661425] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b00a8, hex: 0000000000000000, int: 0},idx=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b00e0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0118, hex: 58A3C56A4A280BE8, int: -1726241732082425000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b0150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-19590, column_count:10, rowkey_column_count:5, has_column_checksum:0, row_count:224, row_store_type:0, opt:0, var_column_count:0, row_offset:15527, original_length:16363, max_merged_trans_version:1726241777443612000, data_length:16363, data_zlength:16363, data_checksum:2057046089, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926f204090, buf_size:16363, data_size:16363, row_count:224, column_count:10, max_merged_trans_version:1726241777443612000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1006468742, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16363}) [2024-09-26 16:01:05.661438] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=13][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:05.661441] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:05.661442] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:05.661443] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:01:05.661445] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:05.661446] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:05.661486] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:01:05.661496] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=6] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.661447] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fc360, local_datums:0x7f92665fc360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fc360, local_datums:0x7f92665fc360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f9269ffe118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269ffe118, limit_param:null, stmt_allocator:0x7f9269ffe7d8, allocator:0x7f9269ffe7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fc360, local_datums:0x7f92665fc360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337665635180, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}) [2024-09-26 16:01:05.661523] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=75][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:05.661525] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fc360, local_datums:0x7f92665fc360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f9269ffe118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9269ffe118, limit_param:null, stmt_allocator:0x7f9269ffe7d8, allocator:0x7f9269ffe7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fc360, local_datums:0x7f92665fc360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fc3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337665635180, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}]) [2024-09-26 16:01:05.661561] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=36][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:05.661564] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=2] partition merge iter row count(i=0, row_count=224, ghost_row_count=0, pkey={tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, table={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337665635180, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.661577] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:05.661596] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:05.661601] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=5][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:05.661603] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337665654765, end_time=1727337665661602, runtime=6837, *this={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72F-0-0, dag_ret:0, dag_status:2, start_time:1727337665635426, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:01:05.661610] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72F-0-0, dag_ret:0, dag_status:2, start_time:1727337665635426, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.661618] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=26191, dag_cnt=41, dag_cnts_[dag.get_type()]=41, &dag=0x7f9264d77a20, dag={ObIDag:{this:0x7f9264d77a20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665635426, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}) [2024-09-26 16:01:05.661626] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72F-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665635424, task_id:YB42C0A8002D-0006230056D8C72F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.661644] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665661641, task_id:YB42C0A8002D-0006230056D8C734-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.661658] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=13] schedule one task(task=0x7f9264d797d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.661682] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.661687] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.661690] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.661686] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.661695] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.661698] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.661695] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=9] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337665635046, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=18, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:01:05.661704] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.661702] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=6] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.661895] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:01:05.661903] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=6] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.661911] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200240}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.661931] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:200240}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200240}, table_count:2, [{i:0, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8637b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665635516, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=26.39ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.662169] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=21] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:01:05.662620] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72B-0-0, dag_ret:0, dag_status:2, start_time:1727337665635398, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5460, output_block_cnt_arr:0x7f9287db5468, concurrent_cnt:1, estimate_row_cnt:85417, estimate_occupy_size:134217728, latest_update_ts:1727337665635516, estimated_finish_time:1727337682662683}) [2024-09-26 16:01:05.662633] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=12] succeed to init merge ctx(task={this:0x7f9264d76920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72B-0-0, dag_ret:0, dag_status:2, start_time:1727337665635398, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.662638] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=5] task finish process(ret=0, start_time=1727337665635514, end_time=1727337665662638, runtime=27124, *this={this:0x7f9264d76920, type:15, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72B-0-0, dag_ret:0, dag_status:2, start_time:1727337665635398, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.662658] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337665635231, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.662680] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=5] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.662670] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828761605424, storage_schema:{this:0x7f9262c77330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828761605424, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.662698] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=23] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.662701] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.662707] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9262c77330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200004}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1710828761605424, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.662726] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:200004}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828761605424, storage_schema:{this:0x7f9262c77330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200004}, table_count:1, [{i:0, table_key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8621da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665656014, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=6.69ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.662750] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C732-0-0, dag_ret:0, dag_status:2, start_time:1727337665655983, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9262c77460, output_block_cnt_arr:0x7f9262c77468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337665656014, estimated_finish_time:1727337682662753}) [2024-09-26 16:01:05.662744] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c6e060, count:3, allocator:0x7f92b33d87d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9262c6e320, local_datums:0x7f9262c6e320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:01:05.662756] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d78d70, type:15, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C732-0-0, dag_ret:0, dag_status:2, start_time:1727337665655983, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.662762] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=5] task finish process(ret=0, start_time=1727337665656012, end_time=1727337665662762, runtime=6750, *this={this:0x7f9264d78d70, type:15, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C732-0-0, dag_ret:0, dag_status:2, start_time:1727337665655983, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.662763] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=16] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fe060, count:3, allocator:0x7f92b33d9558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665fe320, local_datums:0x7f92665fe320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:01:05.662789] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337665635246, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.662797] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:0, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.662806] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.662808] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.662812] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:0, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.662826] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:1}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:0, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1}, table_count:1, [{i:0, table_key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}, merge_scn:{val:1726241621541056000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665656057, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=6.75ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.662842] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C733-0-0, dag_ret:0, dag_status:2, start_time:1727337665655991, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad480, output_block_cnt_arr:0x7f92684ad488, concurrent_cnt:1, estimate_row_cnt:3, estimate_occupy_size:6291456, latest_update_ts:1727337665656057, estimated_finish_time:1727337682662845}) [2024-09-26 16:01:05.662848] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d792a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C733-0-0, dag_ret:0, dag_status:2, start_time:1727337665655991, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:01:05.662852] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=4] task finish process(ret=0, start_time=1727337665656056, end_time=1727337665662852, runtime=6796, *this={this:0x7f9264d792a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C733-0-0, dag_ret:0, dag_status:2, start_time:1727337665655991, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:01:05.662976] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=18][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.663020] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.663023] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf2041c8 0xf217c49 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.663031] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.663033] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.663035] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92665b0108, hex: 313030303030333830, cstr: 100000380},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b00a8, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b00e0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-29413, column_count:20, rowkey_column_count:3, has_column_checksum:0, row_count:100, row_store_type:0, opt:0, var_column_count:0, row_offset:16082, original_length:16422, max_merged_trans_version:1726189672100745000, data_length:16454, data_zlength:5272, data_checksum:4171874375, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926846c030, buf_size:5272, data_size:16454, row_count:100, column_count:20, max_merged_trans_version:1726189672100745000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:371002229, row_count_delta:22, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16422}) [2024-09-26 16:01:05.663046] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=10][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:05.663047] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:05.663050] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:05.663052] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:05.663053] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1526) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, partition_fuser_->get_result_row()={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92664ee148, local_datums:0x7f92664ee148}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee190, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee1c8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3c0, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ee3f8, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, fuse_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9262c6e320, local_datums:0x7f9262c6e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92b33d8118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b33d8118, limit_param:null, stmt_allocator:0x7f92b33d87d8, allocator:0x7f92b33d87d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:-1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9262c6e320, local_datums:0x7f9262c6e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665fe320, local_datums:0x7f92665fe320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92b33d8e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b33d8e98, limit_param:null, stmt_allocator:0x7f92b33d9558, allocator:0x7f92b33d9558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:94314390480640, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665fe320, local_datums:0x7f92665fe320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337665635046, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:01:05.663180] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=126][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9262c6e320, local_datums:0x7f9262c6e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92b33d8118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b33d8118, limit_param:null, stmt_allocator:0x7f92b33d87d8, allocator:0x7f92b33d87d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:-1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9262c6e320, local_datums:0x7f9262c6e320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9262c6e5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665fe320, local_datums:0x7f92665fe320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92b33d8e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b33d8e98, limit_param:null, stmt_allocator:0x7f92b33d9558, allocator:0x7f92b33d9558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:94314390480640, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665fe320, local_datums:0x7f92665fe320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe3a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe5d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337665635046, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:01:05.663270] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=90][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:05.663273] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=2] partition merge iter row count(i=0, row_count=23, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.663284] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=8] partition merge iter row count(i=1, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337665635046, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.663295] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:05.663306] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:05.663309] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:05.663311] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337665662696, end_time=1727337665663311, runtime=615, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72B-0-0, dag_ret:0, dag_status:2, start_time:1727337665635398, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.663318] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72B-0-0, dag_ret:0, dag_status:2, start_time:1727337665635398, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.663326] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=27928, dag_cnt=40, dag_cnts_[dag.get_type()]=40, &dag=0x7f9264d76560, dag={ObIDag:{this:0x7f9264d76560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C72B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665635398, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.663334] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C72B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665635388, task_id:YB42C0A8002D-0006230056D8C72B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.663357] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=6] schedule one task(task=0x7f9264d84ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.663362] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=4] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.663367] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665663365, task_id:YB42C0A8002D-0006230056D8C735-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.663370] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2] schedule one task(task=0x7f9264d79d00, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.663416] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=200004, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:3, allocator:0x7f92864107d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:05.663429] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=1, range=0x7f92aac56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:8}, cols_desc:{ObIArray:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b04060, count:5, allocator:0x7f92636827d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b04360, local_datums:0x7f9265b04360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:05.663576] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.663618] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.663621] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.663627] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.663629] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.663631] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a78070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a780a8, hex: C852A9F488570BE8, int: -1726189786434547000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265a780e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19071, column_count:8, rowkey_column_count:3, has_column_checksum:0, row_count:218, row_store_type:0, opt:0, var_column_count:0, row_offset:15549, original_length:16361, max_merged_trans_version:1726190661653507000, data_length:16361, data_zlength:5996, data_checksum:2604580500, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9287dce030, buf_size:5996, data_size:16361, row_count:218, column_count:8, max_merged_trans_version:1726190661653507000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:83488362, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16361}) [2024-09-26 16:01:05.663641] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=9][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:05.663642] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:05.663644] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:05.663645] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:05.663648] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=3][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f9286410118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286410118, limit_param:null, stmt_allocator:0x7f92864107d8, allocator:0x7f92864107d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337665635231, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:01:05.663699] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=51][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:05.663701] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f9286410118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9286410118, limit_param:null, stmt_allocator:0x7f92864107d8, allocator:0x7f92864107d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337665635231, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:01:05.663737] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=36][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:05.663740] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=2] partition merge iter row count(i=0, row_count=218, ghost_row_count=0, pkey={tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337665635231, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.663752] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:05.663762] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:05.663765] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:05.663766] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337665663370, end_time=1727337665663766, runtime=396, *this={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C732-0-0, dag_ret:0, dag_status:2, start_time:1727337665655983, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.663774] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C732-0-0, dag_ret:0, dag_status:2, start_time:1727337665655983, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.663782] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=7798, dag_cnt=39, dag_cnts_[dag.get_type()]=39, &dag=0x7f9264d789b0, dag={ObIDag:{this:0x7f9264d789b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C732-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665655983, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.663789] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C732-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665655980, task_id:YB42C0A8002D-0006230056D8C732-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.663839] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=12][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.663886] ERROR issue_dba_error (ob_log.cpp:1875) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.663890] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.663897] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.663899] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.663900] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 17, flag: 0, null: 0, ptr: 0x7f9265b0c178, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c0a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 15, flag: 0, null: 0, ptr: 0x7f9265b0c189, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c118, hex: 60D2640CCC290BE8, int: -1726240075808452000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b0c150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-20032, column_count:8, rowkey_column_count:5, has_column_checksum:0, row_count:158, row_store_type:0, opt:0, var_column_count:0, row_offset:15849, original_length:16421, max_merged_trans_version:1726241621541056000, data_length:16421, data_zlength:16421, data_checksum:3599888316, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6c04090, buf_size:16421, data_size:16421, row_count:158, column_count:8, max_merged_trans_version:1726241621541056000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1714289658, row_count_delta:2, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16421}) [2024-09-26 16:01:05.663911] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=11][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:05.663913] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=2][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:05.663915] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:05.663916] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:01:05.663919] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=3][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:05.663921] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:05.663922] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b04360, local_datums:0x7f9265b04360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b043a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b04418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b04450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b04360, local_datums:0x7f9265b04360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b043a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b04418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b04450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f9263682118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263682118, limit_param:null, stmt_allocator:0x7f92636827d8, allocator:0x7f92636827d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b04360, local_datums:0x7f9265b04360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b043a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b04418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b04450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337665635246, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:01:05.663979] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=56][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:05.663980] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b04360, local_datums:0x7f9265b04360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b043a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b04418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b04450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f9263682118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263682118, limit_param:null, stmt_allocator:0x7f92636827d8, allocator:0x7f92636827d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9265b04360, local_datums:0x7f9265b04360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b043a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b04418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b04450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337665635246, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:01:05.664014] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=34][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:05.664016] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=1] partition merge iter row count(i=0, row_count=158, ghost_row_count=0, pkey={tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, table={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337665635246, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.664027] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:05.664036] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:05.664039] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:05.664040] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337665663380, end_time=1727337665664040, runtime=660, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C733-0-0, dag_ret:0, dag_status:2, start_time:1727337665655991, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:01:05.664047] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C733-0-0, dag_ret:0, dag_status:2, start_time:1727337665655991, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.664054] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=8063, dag_cnt=38, dag_cnts_[dag.get_type()]=38, &dag=0x7f9264d78ee0, dag={ObIDag:{this:0x7f9264d78ee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C733-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665655991, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}) [2024-09-26 16:01:05.664061] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C733-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665655988, task_id:YB42C0A8002D-0006230056D8C733-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.664092] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.664095] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.664097] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.664114] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.664152] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=28] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.664172] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.664190] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337665635261, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.664199] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753247280, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828753247280, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.664211] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.664214] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.664219] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:412}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753247280, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.664239] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:412}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753247280, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:412}, table_count:1, [{i:0, table_key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665661677, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.54ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.664262] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C734-0-0, dag_ret:0, dag_status:2, start_time:1727337665661658, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab470, output_block_cnt_arr:0x7f92684ab478, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337665661677, estimated_finish_time:1727337682664264}) [2024-09-26 16:01:05.664269] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d797d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C734-0-0, dag_ret:0, dag_status:2, start_time:1727337665661658, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.664274] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=5] task finish process(ret=0, start_time=1727337665661675, end_time=1727337665664274, runtime=2599, *this={this:0x7f9264d797d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C734-0-0, dag_ret:0, dag_status:2, start_time:1727337665661658, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.664291] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337665635216, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.664300] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=8] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:01:05.664303] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9266505480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.664413] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=6] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3223, total_rows=22561) [2024-09-26 16:01:05.664416] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.664420] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847409}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.664437] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:1152921504606847409}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9266505480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847409}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8640e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665656016, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=8.40ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.664541] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C731-0-0, dag_ret:0, dag_status:2, start_time:1727337665655975, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665055d0, output_block_cnt_arr:0x7f92665055d8, concurrent_cnt:1, estimate_row_cnt:32439, estimate_occupy_size:18874368, latest_update_ts:1727337665656016, estimated_finish_time:1727337682664550}) [2024-09-26 16:01:05.664549] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d78840, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C731-0-0, dag_ret:0, dag_status:2, start_time:1727337665655975, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.664554] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=5] task finish process(ret=0, start_time=1727337665656013, end_time=1727337665664554, runtime=8541, *this={this:0x7f9264d78840, type:15, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C731-0-0, dag_ret:0, dag_status:2, start_time:1727337665655975, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.664599] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2] schedule one task(task=0x7f9268486030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.664604] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=4] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.664611] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665664609, task_id:YB42C0A8002D-0006230056D8C736-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.664615] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=4] schedule one task(task=0x7f9264d7a230, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.664618] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665664618, task_id:YB42C0A8002D-0006230056D8C737-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.664620] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=2] schedule one task(task=0x7f9264d7a760, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.664653] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.664657] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.664659] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.664663] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.664667] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.664673] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241465393532000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.664667] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=1152921504606847409, range=0x7f92aa656200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d2e060, count:5, allocator:0x7f92636367d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9287d2e360, local_datums:0x7f9287d2e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:01:05.664730] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=412, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:7}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fe060, count:4, allocator:0x7f92636867d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f92665fe350, local_datums:0x7f92665fe350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:01:05.664753] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=12][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.664788] ERROR issue_dba_error (ob_log.cpp:1875) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.664791] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.664796] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=5][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.664798] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.664799] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665a6070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 23, flag: 0, null: 0, ptr: 0x7f92665a6140, hex: 6865617274626561745F736572766963655F65706F6368, cstr: heartbeat_service_epoch},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665a60e0, hex: 8808AF9C9C5C0BE8, int: -1726184204453083000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665a6118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-7627, column_count:7, rowkey_column_count:4, has_column_checksum:0, row_count:1, row_store_type:0, opt:5, var_column_count:0, row_offset:136, original_length:80, max_merged_trans_version:1726184204453083000, data_length:80, data_zlength:80, data_checksum:1549639079, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f926f604090, buf_size:80, data_size:80, row_count:1, column_count:7, max_merged_trans_version:1726184204453083000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:37835160, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:80}) [2024-09-26 16:01:05.664811] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:05.664813] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=2][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:01:05.664815] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=1][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:01:05.664816] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=1][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:01:05.664818] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=1][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:01:05.664819] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337665635261, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.664832] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=11][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:05.664841] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:05.664844] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:05.664846] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337665664689, end_time=1727337665664846, runtime=157, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C734-0-0, dag_ret:0, dag_status:2, start_time:1727337665661658, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.664852] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C734-0-0, dag_ret:0, dag_status:2, start_time:1727337665661658, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.664859] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=3201, dag_cnt=37, dag_cnts_[dag.get_type()]=37, &dag=0x7f9264d79410, dag={ObIDag:{this:0x7f9264d79410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C734-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665661658, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.664866] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C734-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665661641, task_id:YB42C0A8002D-0006230056D8C734-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.664900] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.664902] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.664904] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.664909] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.664912] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.664917] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241316899724000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.664922] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=12][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.664978] ERROR issue_dba_error (ob_log.cpp:1875) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.664981] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.664987] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.664989] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.664991] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 7, flag: 0, null: 0, ptr: 0x7f9264a5e178, hex: 31303033343330, cstr: 1003430},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9264a5e17f, hex: 3230323430393133303435303537, cstr: 20240913045057},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e0e0, hex: 6E48500300000000, int: 55593070},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e118, hex: 58B03BC5A1610BE8, int: -1726178684739801000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a5e150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-26357, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:250, row_store_type:0, opt:5, var_column_count:0, row_offset:15394, original_length:16334, max_merged_trans_version:1726190643638412000, data_length:16334, data_zlength:5786, data_checksum:2846467531, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9264a60030, buf_size:5786, data_size:16334, row_count:250, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:796410710, row_count_delta:250, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16334}) [2024-09-26 16:01:05.665002] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:05.665004] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:05.665005] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:05.665007] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:05.665008] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9287d2e360, local_datums:0x7f9287d2e360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9287d2e360, local_datums:0x7f9287d2e360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9263636118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263636118, limit_param:null, stmt_allocator:0x7f92636367d8, allocator:0x7f92636367d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9287d2e360, local_datums:0x7f9287d2e360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337665635216, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:01:05.665058] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=49][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:05.665083] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:05.665059] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847409}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9287d2e360, local_datums:0x7f9287d2e360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9263636118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263636118, limit_param:null, stmt_allocator:0x7f92636367d8, allocator:0x7f92636367d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:3, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178684739801000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9287d2e360, local_datums:0x7f9287d2e360}, datums:[col_id=0:{len: 7, flag: 0, null: 0, ptr: 0x7f927cc5503c, hex: 31303033343330, cstr: 1003430},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f927cc55043, hex: 3230323430393133303435303537, cstr: 20240913045057},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e3e0, hex: 6E48500300000000, int: 55593070},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e418, hex: 58B03BC5A1610BE8, int: -1726178684739801000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d2e450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"1003430", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045057", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55593070}, value_iter:{value:{this=0x7f927cc54f38 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f927cc54fc8 latest_compact_node=(nil) max_trans_version={val:1726178684739801000, v:0} max_trans_id=125096007 max_elr_trans_version={val:1726178684739801000, v:0} max_elr_trans_id=125096007 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178684739801000, v:0} min_modify_scn={val:1726178684739801000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178684739801000, cur_trans_version:{val:1726178684739801000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:250, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337665635216, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:01:05.665112] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=53][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:05.665115] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2] partition merge iter row count(i=0, row_count=250, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337665635216, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.665130] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=10][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:05.665141] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:05.665144] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:05.665164] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=20] task finish process(ret=-4184, start_time=1727337665664633, end_time=1727337665665164, runtime=531, *this={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C731-0-0, dag_ret:0, dag_status:2, start_time:1727337665655975, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.665171] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9268486030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C731-0-0, dag_ret:0, dag_status:2, start_time:1727337665655975, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.665179] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=9204, dag_cnt=36, dag_cnts_[dag.get_type()]=36, &dag=0x7f9264d78480, dag={ObIDag:{this:0x7f9264d78480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C731-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665655975, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.665187] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C731-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665655970, task_id:YB42C0A8002D-0006230056D8C731-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.665224] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337665635289, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.665233] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753327496, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753327496, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.665244] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.665247] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.665251] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:444}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753327496, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.665268] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:444}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753327496, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:444}, table_count:1, [{i:0, table_key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8632d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665664086, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.16ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.665289] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C735-0-0, dag_ret:0, dag_status:2, start_time:1727337665663370, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1490, output_block_cnt_arr:0x7f92728f1498, concurrent_cnt:1, estimate_row_cnt:4, estimate_occupy_size:6291456, latest_update_ts:1727337665664086, estimated_finish_time:1727337682665292}) [2024-09-26 16:01:05.665295] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d79d00, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C735-0-0, dag_ret:0, dag_status:2, start_time:1727337665663370, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.665299] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=4] task finish process(ret=0, start_time=1727337665664084, end_time=1727337665665299, runtime=1215, *this={this:0x7f9264d79d00, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C735-0-0, dag_ret:0, dag_status:2, start_time:1727337665663370, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.665363] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.665370] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337665665369, task_id:YB42C0A8002D-0006230056D8C738-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.665374] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=3] schedule one task(task=0x7f9264d7ac90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.665379] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337665665378, task_id:YB42C0A8002D-0006230056D8C739-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.665382] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=2] schedule one task(task=0x7f9264d7b1c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.665393] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=32] success to notify_need_writing_throttling True [2024-09-26 16:01:05.665397] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=2][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.665422] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=444, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ec060, count:6, allocator:0x7f92865a27d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664ec390, local_datums:0x7f92664ec390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:01:05.665610] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=13][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.665647] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.665649] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.665655] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.665657] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.665659] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f00a8, hex: 0300000000000000, int: 3},idx=2:{len: 12, flag: 0, null: 0, ptr: 0x7f92664f01b0, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0118, hex: 420B000000000000, int: 2882},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0150, hex: 288E681FE7280BE8, int: -1726241059036951000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0188, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:24981, column_count:9, rowkey_column_count:6, has_column_checksum:0, row_count:207, row_store_type:0, opt:0, var_column_count:0, row_offset:15610, original_length:16378, max_merged_trans_version:1726241059036951000, data_length:16378, data_zlength:16378, data_checksum:1574137226, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6604090, buf_size:16378, data_size:16378, row_count:207, column_count:9, max_merged_trans_version:1726241059036951000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:666643265, row_count_delta:3, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16378}) [2024-09-26 16:01:05.665670] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=11][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:05.665672] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=1][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:05.665673] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:05.665674] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:01:05.665676] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:05.665677] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:05.665679] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664ec390, local_datums:0x7f92664ec390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664ec390, local_datums:0x7f92664ec390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f92865a2118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92865a2118, limit_param:null, stmt_allocator:0x7f92865a27d8, allocator:0x7f92865a27d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664ec390, local_datums:0x7f92664ec390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337665635289, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}) [2024-09-26 16:01:05.665733] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=54][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:05.665735] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664ec390, local_datums:0x7f92664ec390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f92865a2118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92865a2118, limit_param:null, stmt_allocator:0x7f92865a27d8, allocator:0x7f92865a27d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664ec390, local_datums:0x7f92664ec390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec3d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec4b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664ec560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337665635289, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}]) [2024-09-26 16:01:05.665772] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=37][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:05.665774] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=1] partition merge iter row count(i=0, row_count=207, ghost_row_count=0, pkey={tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337665635289, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.665784] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:05.665792] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:05.665794] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:05.665796] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337665665377, end_time=1727337665665796, runtime=419, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C735-0-0, dag_ret:0, dag_status:2, start_time:1727337665663370, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.665802] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C735-0-0, dag_ret:0, dag_status:2, start_time:1727337665663370, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.665809] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=2439, dag_cnt=35, dag_cnts_[dag.get_type()]=35, &dag=0x7f9264d79940, dag={ObIDag:{this:0x7f9264d79940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C735-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665663370, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.665816] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C735-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665663365, task_id:YB42C0A8002D-0006230056D8C735-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.665845] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=7][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.665848] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.665850] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.665854] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.665857] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.665862] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.665883] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.665885] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.665886] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.665889] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.665891] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.665895] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.665907] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337665635305, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.665916] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752918752, storage_schema:{this:0x7f9287ddb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828752918752, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.665928] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.665931] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.665936] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287ddb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:309}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752918752, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.665954] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:309}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752918752, storage_schema:{this:0x7f9287ddb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:309}, table_count:1, [{i:0, table_key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}, merge_scn:{val:1726241316899724000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8631da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665664895, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.04ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.665977] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C736-0-0, dag_ret:0, dag_status:2, start_time:1727337665664615, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}, scanned_row_cnt_arr:0x7f9287ddb490, output_block_cnt_arr:0x7f9287ddb498, concurrent_cnt:1, estimate_row_cnt:6, estimate_occupy_size:6291456, latest_update_ts:1727337665664895, estimated_finish_time:1727337682665980}) [2024-09-26 16:01:05.665983] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7a230, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C736-0-0, dag_ret:0, dag_status:2, start_time:1727337665664615, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:01:05.665988] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=4] task finish process(ret=0, start_time=1727337665664893, end_time=1727337665665988, runtime=1095, *this={this:0x7f9264d7a230, type:15, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C736-0-0, dag_ret:0, dag_status:2, start_time:1727337665664615, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:01:05.666908] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337665635320, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.666918] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.666927] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.666929] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.666938] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=1] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.666934] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:331}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.666949] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337665666947, task_id:YB42C0A8002D-0006230056D8C73A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.666954] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=5] schedule one task(task=0x7f9264d7b6f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.666968] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:331}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92684b1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:331}, table_count:1, [{i:0, table_key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}, merge_scn:{val:1726241465393532000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8633ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665664648, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.28ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.666991] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C737-0-0, dag_ret:0, dag_status:2, start_time:1727337665664620, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}, scanned_row_cnt_arr:0x7f92684b1480, output_block_cnt_arr:0x7f92684b1488, concurrent_cnt:1, estimate_row_cnt:45, estimate_occupy_size:0, latest_update_ts:1727337665664648, estimated_finish_time:1727337682666991}) [2024-09-26 16:01:05.666998] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7a760, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C737-0-0, dag_ret:0, dag_status:2, start_time:1727337665664620, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:01:05.667004] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=6] task finish process(ret=0, start_time=1727337665664646, end_time=1727337665667004, runtime=2358, *this={this:0x7f9264d7a760, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C737-0-0, dag_ret:0, dag_status:2, start_time:1727337665664620, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:01:05.667037] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=3] schedule one task(task=0x7f9287d5e3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.667039] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.667042] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.667054] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=12][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.667075] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=19] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.667079] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.667084] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.667134] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=309, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:11}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:7, allocator:0x7f92864427d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f92664f03b0, local_datums:0x7f92664f03b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:01:05.667297] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=13][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.667351] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.667355] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.667362] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.667367] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=5][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.667369] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:7, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 12, flag: 0, null: 0, ptr: 0x7f92665fe1e8, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe0e0, hex: 420B000000000000, int: 2882},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe118, hex: 0100000000000000, int: 1, num_digit0: 0},idx=4:{len: 16, flag: 0, null: 0, ptr: 0x7f92665fe1f4, hex: 74656E616E7420636C6F672064617461, cstr: tenant clog data},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe188, hex: C818154EB7520BE8, int: -1726195084923955000},idx=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fe1c0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19292, column_count:11, rowkey_column_count:7, has_column_checksum:0, row_count:156, row_store_type:0, opt:0, var_column_count:0, row_offset:15827, original_length:16391, max_merged_trans_version:1726241316783508000, data_length:16391, data_zlength:16391, data_checksum:3629777942, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926f804090, buf_size:16391, data_size:16391, row_count:156, column_count:11, max_merged_trans_version:1726241316783508000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:619385244, row_count_delta:1, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16391}) [2024-09-26 16:01:05.667381] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=11][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:05.667382] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=1][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.667386] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=1] partition merge iter row count(i=0, row_count=156, ghost_row_count=0, pkey={tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, table={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337665635305, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.667406] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665667099, end_time=1727337665667405, runtime=306, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C736-0-0, dag_ret:0, dag_status:2, start_time:1727337665664615, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:01:05.667415] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2800, dag_cnt=34, dag_cnts_[dag.get_type()]=34, &dag=0x7f9264d79e70, dag={ObIDag:{this:0x7f9264d79e70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C736-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665664615, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}) [2024-09-26 16:01:05.667423] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C736-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337665664609, task_id:YB42C0A8002D-0006230056D8C736-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.667452] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337665667436, task_id:YB42C0A8002D-0006230056D8C73B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.667458] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=5] schedule one task(task=0x7f9264d7bc20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.667506] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=331, range=0x7f92a9b52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d34060, count:5, allocator:0x7f92b32407d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:01:05.667698] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.667734] ERROR issue_dba_error (ob_log.cpp:1875) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.667737] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.667745] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=0] partition merge iter row count(i=0, row_count=196, ghost_row_count=0, pkey={tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, table={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337665635320, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.667762] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665667467, end_time=1727337665667762, runtime=295, *this={this:0x7f9287d5e3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C737-0-0, dag_ret:0, dag_status:2, start_time:1727337665664620, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:01:05.667770] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3150, dag_cnt=33, dag_cnts_[dag.get_type()]=33, &dag=0x7f9264d7a3a0, dag={ObIDag:{this:0x7f9264d7a3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C737-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665664620, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}) [2024-09-26 16:01:05.667777] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C737-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665664618, task_id:YB42C0A8002D-0006230056D8C737-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.667805] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.667803] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337665667790, task_id:YB42C0A8002D-0006230056D8C73C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.667811] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.667813] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.667816] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.667820] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.667818] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=14] schedule one task(task=0x7f9264d7c150, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.667825] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.667847] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.667851] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.667847] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337665635349, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.667853] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.667858] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.667855] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753417144, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828753417144, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.667861] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.667869] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.667927] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:01:05.667932] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.667938] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:487}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753417144, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.667968] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=25] succeed to build merge ctx(tablet_id={id:487}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753417144, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:487}, table_count:1, [{i:0, table_key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665665880, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.06ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.668070] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=20] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:01:05.668180] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C739-0-0, dag_ret:0, dag_status:2, start_time:1727337665665382, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f4a0, output_block_cnt_arr:0x7f9264b1f4a8, concurrent_cnt:1, estimate_row_cnt:2199, estimate_occupy_size:0, latest_update_ts:1727337665665880, estimated_finish_time:1727337682668180}) [2024-09-26 16:01:05.668189] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d7b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C739-0-0, dag_ret:0, dag_status:2, start_time:1727337665665382, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.668194] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=4] task finish process(ret=0, start_time=1727337665665877, end_time=1727337665668194, runtime=2317, *this={this:0x7f9264d7b1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C739-0-0, dag_ret:0, dag_status:2, start_time:1727337665665382, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.668223] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337665635335, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.668232] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753357664, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753357664, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.668287] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=7] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:01:05.668292] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.668297] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:458}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753357664, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.668314] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:458}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753357664, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:458}, table_count:1, [{i:0, table_key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8627b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665665841, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.45ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.668343] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337665635553, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.668360] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=16] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753441792, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828753441792, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.668413] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=18] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:01:05.668421] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1295, total_rows=1295) [2024-09-26 16:01:05.668425] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.668431] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:501}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753441792, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.668452] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:501}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753441792, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:501}, table_count:1, [{i:0, table_key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665667841, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=588us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.668530] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73C-0-0, dag_ret:0, dag_status:2, start_time:1727337665667817, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b21490, output_block_cnt_arr:0x7f9264b21498, concurrent_cnt:1, estimate_row_cnt:1295, estimate_occupy_size:0, latest_update_ts:1727337665667841, estimated_finish_time:1727337682668529}) [2024-09-26 16:01:05.668537] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7c150, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73C-0-0, dag_ret:0, dag_status:2, start_time:1727337665667817, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.668545] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=7] task finish process(ret=0, start_time=1727337665667838, end_time=1727337665668544, runtime=706, *this={this:0x7f9264d7c150, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73C-0-0, dag_ret:0, dag_status:2, start_time:1727337665667817, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.668554] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C738-0-0, dag_ret:0, dag_status:2, start_time:1727337665665374, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d4a0, output_block_cnt_arr:0x7f9264b1d4a8, concurrent_cnt:1, estimate_row_cnt:14598, estimate_occupy_size:4194304, latest_update_ts:1727337665665841, estimated_finish_time:1727337682668556}) [2024-09-26 16:01:05.668562] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d7ac90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C738-0-0, dag_ret:0, dag_status:2, start_time:1727337665665374, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.668567] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=5] schedule one task(task=0x7f9264afa030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.668567] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=5] task finish process(ret=0, start_time=1727337665665840, end_time=1727337665668567, runtime=2727, *this={this:0x7f9264d7ac90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C738-0-0, dag_ret:0, dag_status:2, start_time:1727337665665374, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.668572] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=4] schedule one task(task=0x7f9264b56030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.668605] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337665635394, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.668615] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753437040, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=57, max_schema_version_in_memtable=1710828753437040, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.668648] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=487, range=0x7f92a9856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:11}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5c060, count:8, allocator:0x7f92636847d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9263b5c3e0, local_datums:0x7f9263b5c3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:01:05.668673] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1362, total_rows=1362) [2024-09-26 16:01:05.668676] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.668683] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:492}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:12, rowkey_column_count:11, schema_rowkey_col_cnt:9, full_stored_col_cnt:59, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753437040, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=28 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.668705] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=18] succeed to build merge ctx(tablet_id={id:492}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753437040, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:492}, table_count:1, [{i:0, table_key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8624c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665667801, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=880us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.668776] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73B-0-0, dag_ret:0, dag_status:2, start_time:1727337665667457, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a434c0, output_block_cnt_arr:0x7f9263a434c8, concurrent_cnt:1, estimate_row_cnt:1362, estimate_occupy_size:2097152, latest_update_ts:1727337665667801, estimated_finish_time:1727337682668777}) [2024-09-26 16:01:05.668782] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7bc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73B-0-0, dag_ret:0, dag_status:2, start_time:1727337665667457, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.668786] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=4] task finish process(ret=0, start_time=1727337665667800, end_time=1727337665668786, runtime=986, *this={this:0x7f9264d7bc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73B-0-0, dag_ret:0, dag_status:2, start_time:1727337665667457, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.668841] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337665635376, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.668850] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753332736, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=36, max_schema_version_in_memtable=1710828753332736, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.668892] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.668931] ERROR issue_dba_error (ob_log.cpp:1875) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.668934] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.668945] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=0] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337665635349, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.668969] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665668604, end_time=1727337665668969, runtime=365, *this={this:0x7f9264afa030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C739-0-0, dag_ret:0, dag_status:2, start_time:1727337665665382, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.668980] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3598, dag_cnt=32, dag_cnts_[dag.get_type()]=32, &dag=0x7f9264d7ae00, dag={ObIDag:{this:0x7f9264d7ae00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C739-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665665382, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.668984] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=7] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:01:05.668988] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C739-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665665378, task_id:YB42C0A8002D-0006230056D8C739-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.669029] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337665355017]}) [2024-09-26 16:01:05.669037] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.669043] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:455}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:10, rowkey_column_count:9, schema_rowkey_col_cnt:7, full_stored_col_cnt:38, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753332736, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=26 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.669063] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:455}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753332736, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:455}, table_count:1, [{i:0, table_key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665667035, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.00ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.669072] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=501, range=0x7f92aaed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:6, allocator:0x7f92c1d9e7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9264a62390, local_datums:0x7f9264a62390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:05.669162] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.669195] ERROR issue_dba_error (ob_log.cpp:1875) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.669198] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.669207] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=0] partition merge iter row count(i=0, row_count=56, ghost_row_count=0, pkey={tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337665635553, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.669221] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=16] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:01:05.669228] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665669038, end_time=1727337665669228, runtime=190, *this={this:0x7f9264b56030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73C-0-0, dag_ret:0, dag_status:2, start_time:1727337665667817, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.669237] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1420, dag_cnt=31, dag_cnts_[dag.get_type()]=31, &dag=0x7f9264d7bd90, dag={ObIDag:{this:0x7f9264d7bd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665667817, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.669245] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73C-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337665667790, task_id:YB42C0A8002D-0006230056D8C73C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.669341] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=4] schedule one task(task=0x7f9264d84ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.669350] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337665669348, task_id:YB42C0A8002D-0006230056D8C73D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.669353] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=3] schedule one task(task=0x7f9264d7c680, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.669357] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665669357, task_id:YB42C0A8002D-0006230056D8C73E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.669359] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=2] schedule one task(task=0x7f9264d7cbb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.669389] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.669393] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.669395] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.669400] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.669429] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73A-0-0, dag_ret:0, dag_status:2, start_time:1727337665666954, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db34b0, output_block_cnt_arr:0x7f9287db34b8, concurrent_cnt:1, estimate_row_cnt:3824, estimate_occupy_size:4194304, latest_update_ts:1727337665667035, estimated_finish_time:1727337682669431}) [2024-09-26 16:01:05.669432] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.669438] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.669438] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264d7b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73A-0-0, dag_ret:0, dag_status:2, start_time:1727337665666954, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.669441] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726077706339704027) [2024-09-26 16:01:05.669446] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.669443] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=5] task finish process(ret=0, start_time=1727337665667033, end_time=1727337665669443, runtime=2410, *this={this:0x7f9264d7b6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73A-0-0, dag_ret:0, dag_status:2, start_time:1727337665666954, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.669404] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b862a0a0, is_inited:true, wash_score:1727337665635404000, hold_ref_cnt:true, ref_cnt:2, version:2, length:490, tablet_addr:[816017-1849-0-2305843629838340096-2](offset=1871872,size=578,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:472}, data_tablet_id:{id:472}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:19976489, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862a734, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844179594129408-0](offset=1847296,size=1090,type=2,seq=0), type:"MAJOR", tablet_id:{id:472}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389523010000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843910084932674-0](offset=1848386,size=839,type=2,seq=0), type:"MINOR", tablet_id:{id:472}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178194887823000, upper_trans_version:1726178194887823000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337665635573, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd4080, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209505593}, this:0x7f92a3bd4080, timestamp:1727334209505593, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:65, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274254072414000, v:0}, rec_scn:{val:1726241829658678000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697839104-2](offset=1859584,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834376524-0](offset=1859916,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881206236-0](offset=1860060,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447726270-0](offset=1860286,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7ac60, 0x7f92a3bd4080, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:05.669450] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.669481] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.669484] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=73] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.669499] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.669547] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=2] schedule one task(task=0x7f9264d85260, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.669539] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=492, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:57, schema_rowkey_cnt:9, rowkey_cnt:11, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:9, column_cnt:59}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:11, allocator:0x7f928642a7d8, capacity:11, init_cnt:11}, datum_utils:{is_oracle_mode:false, rowkey_cnt:11, col_cnt:11, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:59, datum_buffer:{capacity:59, datums:0x7f9263b5e050, local_datums:0x7f9263b5a440}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,col_id=43:NOP,col_id=44:NOP,col_id=45:NOP,col_id=46:NOP,col_id=47:NOP,col_id=48:NOP,col_id=49:NOP,col_id=50:NOP,col_id=51:NOP,col_id=52:NOP,col_id=53:NOP,col_id=54:NOP,col_id=55:NOP,col_id=56:NOP,col_id=57:NOP,col_id=58:NOP,]}) [2024-09-26 16:01:05.669616] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=455, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:36, schema_rowkey_cnt:7, rowkey_cnt:9, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:7, column_cnt:38}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263be2060, count:9, allocator:0x7f92636807d8, capacity:9, init_cnt:9}, datum_utils:{is_oracle_mode:false, rowkey_cnt:9, col_cnt:9, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:38, datum_buffer:{capacity:38, datums:0x7f9263be3340, local_datums:0x7f9263be23f0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,]}) [2024-09-26 16:01:05.669763] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=28][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.669801] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.669804] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.669813] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=0] partition merge iter row count(i=0, row_count=61, ghost_row_count=0, pkey={tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337665635394, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.669833] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665669483, end_time=1727337665669833, runtime=350, *this={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73B-0-0, dag_ret:0, dag_status:2, start_time:1727337665667457, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.669841] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2384, dag_cnt=30, dag_cnts_[dag.get_type()]=30, &dag=0x7f9264d7b860, dag={ObIDag:{this:0x7f9264d7b860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665667457, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.669850] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73B-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337665667436, task_id:YB42C0A8002D-0006230056D8C73B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.669856] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=21][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.669897] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.669900] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.669911] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=0] partition merge iter row count(i=0, row_count=140, ghost_row_count=0, pkey={tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337665635376, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.669936] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665669571, end_time=1727337665669936, runtime=365, *this={this:0x7f9264d85260, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73A-0-0, dag_ret:0, dag_status:2, start_time:1727337665666954, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.669947] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2992, dag_cnt=29, dag_cnts_[dag.get_type()]=29, &dag=0x7f9264d7b330, dag={ObIDag:{this:0x7f9264d7b330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665666954, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.669955] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73A-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665666947, task_id:YB42C0A8002D-0006230056D8C73A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.670026] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337665635197, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.670037] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=10] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:01:05.670040] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f9263b57480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.670162] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=18] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3214, total_rows=22498) [2024-09-26 16:01:05.670165] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.670171] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263b57480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847408}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.670187] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:1152921504606847408}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f9263b57480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847408}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665655349, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=14.82ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.670294] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C730-0-0, dag_ret:0, dag_status:2, start_time:1727337665654760, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263b575d0, output_block_cnt_arr:0x7f9263b575d8, concurrent_cnt:1, estimate_row_cnt:35098, estimate_occupy_size:18874368, latest_update_ts:1727337665655349, estimated_finish_time:1727337682670303}) [2024-09-26 16:01:05.670300] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d78310, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C730-0-0, dag_ret:0, dag_status:2, start_time:1727337665654760, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.670305] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=5] task finish process(ret=0, start_time=1727337665655346, end_time=1727337665670305, runtime=14959, *this={this:0x7f9264d78310, type:15, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C730-0-0, dag_ret:0, dag_status:2, start_time:1727337665654760, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.670498] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337665635573, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.670534] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=35] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753360568, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753360568, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.670540] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337665635587, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.670548] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.670552] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.670552] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753001832, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=21, max_schema_version_in_memtable=1710828753001832, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.670582] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=25] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.670559] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:472}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753360568, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.670585] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.670592] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:342}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:23, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753001832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.670591] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=26] succeed to build merge ctx(tablet_id={id:472}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753360568, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:472}, table_count:1, [{i:0, table_key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665669373, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.18ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.670616] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73D-0-0, dag_ret:0, dag_status:2, start_time:1727337665669353, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505460, output_block_cnt_arr:0x7f9266505468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337665669373, estimated_finish_time:1727337682670616}) [2024-09-26 16:01:05.670623] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7c680, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73D-0-0, dag_ret:0, dag_status:2, start_time:1727337665669353, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.670611] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:342}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753001832, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:342}, table_count:1, [{i:0, table_key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665669426, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.16ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.670628] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=5] task finish process(ret=0, start_time=1727337665669372, end_time=1727337665670628, runtime=1256, *this={this:0x7f9264d7c680, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73D-0-0, dag_ret:0, dag_status:2, start_time:1727337665669353, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.670635] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73E-0-0, dag_ret:0, dag_status:2, start_time:1727337665669359, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a45490, output_block_cnt_arr:0x7f9263a45498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337665669426, estimated_finish_time:1727337682670635}) [2024-09-26 16:01:05.670642] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73E-0-0, dag_ret:0, dag_status:2, start_time:1727337665669359, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.670647] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=4] task finish process(ret=0, start_time=1727337665669423, end_time=1727337665670646, runtime=1223, *this={this:0x7f9264d7cbb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73E-0-0, dag_ret:0, dag_status:2, start_time:1727337665669359, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.670663] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=3] schedule one task(task=0x7f9264d84ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.670667] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=3] schedule one task(task=0x7f9264b36030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.670670] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=2] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.670673] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=2] schedule one task(task=0x7f9264b363c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.670677] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337665670676, task_id:YB42C0A8002D-0006230056D8C73F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.670681] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=4] schedule one task(task=0x7f9264d7d0e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.670685] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665670684, task_id:YB42C0A8002D-0006230056D8C740-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.670687] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=1] schedule one task(task=0x7f9264d7d610, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.670708] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=472, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d58060, count:3, allocator:0x7f928649c7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f9264d58320, local_datums:0x7f9264d58320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:01:05.670719] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=458, range=0x7f92aab52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a64060, count:8, allocator:0x7f92ba7fc7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9264a643e0, local_datums:0x7f9264a643e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:01:05.670744] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.670738] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=342, range=0x7f92ab0d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:21, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:23}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f926589e060, count:6, allocator:0x7f92a31fc7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:23, datum_buffer:{capacity:32, datums:0x7f926589e390, local_datums:0x7f926589e390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,]}) [2024-09-26 16:01:05.670770] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.670779] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.670781] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.670790] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=0] partition merge iter row count(i=0, row_count=17, ghost_row_count=0, pkey={tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337665635573, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.670807] ERROR issue_dba_error (ob_log.cpp:1875) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.670810] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665670680, end_time=1727337665670809, runtime=129, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73D-0-0, dag_ret:0, dag_status:2, start_time:1727337665669353, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.670810] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.670818] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1465, dag_cnt=28, dag_cnts_[dag.get_type()]=28, &dag=0x7f9264d7c2c0, dag={ObIDag:{this:0x7f9264d7c2c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665669353, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.670826] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73D-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337665669348, task_id:YB42C0A8002D-0006230056D8C73D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.670819] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337665635587, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.670852] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337665670699, end_time=1727337665670852, runtime=153, *this={this:0x7f9264b363c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73E-0-0, dag_ret:0, dag_status:2, start_time:1727337665669359, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.670863] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1504, dag_cnt=27, dag_cnts_[dag.get_type()]=27, &dag=0x7f9264d7c7f0, dag={ObIDag:{this:0x7f9264d7c7f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665669359, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.670870] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73E-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665669357, task_id:YB42C0A8002D-0006230056D8C73E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.670923] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=1152921504606847408, range=0x7f92a9952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:5, allocator:0x7f9269ffe7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:01:05.670946] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.670998] ERROR issue_dba_error (ob_log.cpp:1875) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.671001] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.671010] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=0] partition merge iter row count(i=0, row_count=247, ghost_row_count=0, pkey={tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337665635335, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.671029] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665670680, end_time=1727337665671029, runtime=349, *this={this:0x7f9264b36030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C738-0-0, dag_ret:0, dag_status:2, start_time:1727337665665374, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.671038] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5663, dag_cnt=26, dag_cnts_[dag.get_type()]=26, &dag=0x7f9264d7a8d0, dag={ObIDag:{this:0x7f9264d7a8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C738-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665665374, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.671045] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C738-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665665369, task_id:YB42C0A8002D-0006230056D8C738-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.671074] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.671077] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.671079] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.671083] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.671086] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8641da0, is_inited:true, wash_score:1727337665635404000, hold_ref_cnt:true, ref_cnt:2, version:2, length:487, tablet_addr:[816017-1849-0-2305843626617249792-2](offset=2007040,size=575,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101009}, data_tablet_id:{id:140}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726073917016589000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3251381279, row_count:150}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8642458, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844188184215552-0](offset=1998848,size=1098,type=2,seq=0), type:"MAJOR", tablet_id:{id:101009}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726070403667750001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843957329724490-0](offset=1999946,size=883,type=2,seq=0), type:"MINOR", tablet_id:{id:101009}, scn_range:{start_scn:{val:1726073917016589000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726167395874925000, upper_trans_version:1726167395874925000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337665635619, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd0ba0, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209495816}, this:0x7f92a3bd0ba0, timestamp:1727334209495816, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:59, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726256203821911000, v:0}, rec_scn:{val:1726241803398655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697982464-2](offset=2002944,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170276970828-0](offset=2003276,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881349602-0](offset=2003426,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7bc00, 0x7f92a3bd0ba0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:05.671144] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=58] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.671151] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.671173] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.671209] ERROR issue_dba_error (ob_log.cpp:1875) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.671212] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.671220] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=0] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337665635197, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.671243] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665670893, end_time=1727337665671243, runtime=350, *this={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C730-0-0, dag_ret:0, dag_status:2, start_time:1727337665654760, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.671252] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=16492, dag_cnt=25, dag_cnts_[dag.get_type()]=25, &dag=0x7f9264d77f50, dag={ObIDag:{this:0x7f9264d77f50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C730-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665654760, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.671259] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C730-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665654752, task_id:YB42C0A8002D-0006230056D8C730-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.671289] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.671292] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.671294] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.671300] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.671303] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.671310] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.671354] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337665671352, task_id:YB42C0A8002D-0006230056D8C741-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.671357] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=3] schedule one task(task=0x7f9264d7db40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.671362] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337665671361, task_id:YB42C0A8002D-0006230056D8C742-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.671364] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=2] schedule one task(task=0x7f9264d7e070, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.671370] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337665671369, task_id:YB42C0A8002D-0006230056D8C743-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.671373] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=3] schedule one task(task=0x7f9264d7e5a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.671376] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665671375, task_id:YB42C0A8002D-0006230056D8C744-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.671378] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=2] schedule one task(task=0x7f9264d7ead0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.671393] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.671396] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.671396] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.671398] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:05.671399] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.671401] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:05.671401] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.671404] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.671403] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.671410] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.671413] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.671417] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.671419] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:05.671423] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.671427] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.671433] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.671407] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864df20, is_inited:true, wash_score:1727337665635404000, hold_ref_cnt:true, ref_cnt:2, version:2, length:503, tablet_addr:[814042-5586-0-2305843643796283392-2](offset=1171456,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:377}, data_tablet_id:{id:377}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4164021307, row_count:17}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864e5fc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844760487772160-0](offset=1163264,size=1631,type=2,seq=0), type:"MAJOR", tablet_id:{id:377}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563080818000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843989541144159-0](offset=1164895,size=913,type=2,seq=0), type:"MINI", tablet_id:{id:377}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557779816000, upper_trans_version:1726121557779816000}, {i:1, addr:[814042-5586-0-2305843969140050416-0](offset=1165808,size=894,type=2,seq=0), type:"MINI", tablet_id:{id:377}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986451466000, upper_trans_version:1726149986451466000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337665635665, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdb5b0, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220773832}, this:0x7f92a3bdb5b0, timestamp:1727334220773832, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:80, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293334630000, v:0}, rec_scn:{val:1726264800181268000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513780224-2](offset=1167360,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843176718586294-0](offset=1167798,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880514130-0](offset=1167954,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447034164-0](offset=1168180,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7e310, 0x7f92a3bdb5b0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:05.671441] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=34] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.671446] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.671475] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.671478] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.671480] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:05.671483] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.671485] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.671489] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.671518] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337665635619, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.671526] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.671536] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.671539] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.671544] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101009}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:01:05.671561] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:101009}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101009}, table_count:1, [{i:0, table_key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8641da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665671069, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=473us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.671573] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337665635602, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.671586] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C740-0-0, dag_ret:0, dag_status:2, start_time:1727337665670687, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3470, output_block_cnt_arr:0x7f9287db3478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337665671069, estimated_finish_time:1727337682671586}) [2024-09-26 16:01:05.671593] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7d610, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C740-0-0, dag_ret:0, dag_status:2, start_time:1727337665670687, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.671593] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=19] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.671598] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=4] task finish process(ret=0, start_time=1727337665671068, end_time=1727337665671597, runtime=529, *this={this:0x7f9264d7d610, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C740-0-0, dag_ret:0, dag_status:2, start_time:1727337665670687, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.671604] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.671616] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=11] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.671622] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:140}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.671638] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:140}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:140}, table_count:1, [{i:0, table_key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8622d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665671283, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=336us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.671661] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73F-0-0, dag_ret:0, dag_status:2, start_time:1727337665670681, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1460, output_block_cnt_arr:0x7f92728f1468, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337665671283, estimated_finish_time:1727337682671661}) [2024-09-26 16:01:05.671667] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73F-0-0, dag_ret:0, dag_status:2, start_time:1727337665670681, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.671672] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=5] task finish process(ret=0, start_time=1727337665671282, end_time=1727337665671672, runtime=390, *this={this:0x7f9264d7d0e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73F-0-0, dag_ret:0, dag_status:2, start_time:1727337665670681, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.671723] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337665635635, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.671734] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.671742] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.671745] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.671749] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101010}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:01:05.671763] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:101010}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9287db5330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101010}, table_count:1, [{i:0, table_key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8642d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665671390, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=358us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.671782] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C741-0-0, dag_ret:0, dag_status:2, start_time:1727337665671357, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db5470, output_block_cnt_arr:0x7f9287db5478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337665671390, estimated_finish_time:1727337682671781}) [2024-09-26 16:01:05.671787] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7db40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C741-0-0, dag_ret:0, dag_status:2, start_time:1727337665671357, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.671791] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=4] task finish process(ret=0, start_time=1727337665671388, end_time=1727337665671791, runtime=403, *this={this:0x7f9264d7db40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C741-0-0, dag_ret:0, dag_status:2, start_time:1727337665671357, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.671804] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=1] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.671808] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=3] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.671811] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=2] schedule one task(task=0x7f9264d84ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.671855] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101009, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:4, allocator:0x7f92636347d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263b5e350, local_datums:0x7f9263b5e350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:05.671870] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101010, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263be4060, count:4, allocator:0x7f92bc8f47d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9263be4350, local_datums:0x7f9263be4350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:05.671905] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=13][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.671909] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.671949] ERROR issue_dba_error (ob_log.cpp:1875) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.671952] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.671958] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.671959] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.671962] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.671961] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.671963] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 6, flag: 0, null: 0, ptr: 0x7f9264dd2140, hex: 736572766572, cstr: server},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264dd2146, hex: 4BB95CACF5210600, int: 1726188902725963},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264dd20e0, hex: E0A36CB356580BE8, int: -1726188902765780000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264dd2118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-21579, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:20, row_store_type:0, opt:5, var_column_count:0, row_offset:1114, original_length:1134, max_merged_trans_version:1726239607937620000, data_length:1134, data_zlength:1134, data_checksum:3632295672, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92b6804090, buf_size:1134, data_size:1134, row_count:20, column_count:4, max_merged_trans_version:1726239607937620000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3467544488, row_count_delta:20, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:1134}) [2024-09-26 16:01:05.671975] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:05.671977] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=2][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:01:05.671979] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=2][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:01:05.671981] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=1][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:01:05.671972] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337665635635, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.671982] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=1][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:01:05.671984] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=2] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337665635619, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.671998] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=11][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:05.671993] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665671842, end_time=1727337665671993, runtime=151, *this={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C741-0-0, dag_ret:0, dag_status:2, start_time:1727337665671357, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672005] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:05.672004] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=647, dag_cnt=24, dag_cnts_[dag.get_type()]=24, &dag=0x7f9264d7d780, dag={ObIDag:{this:0x7f9264d7d780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C741-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665671357, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.672011] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C741-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665671352, task_id:YB42C0A8002D-0006230056D8C741-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.672022] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=16][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:05.672024] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337665671828, end_time=1727337665672024, runtime=196, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C740-0-0, dag_ret:0, dag_status:2, start_time:1727337665670687, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672031] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C740-0-0, dag_ret:0, dag_status:2, start_time:1727337665670687, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.672054] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=20] dag finished(dag_ret=-4184, runtime=1367, dag_cnt=23, dag_cnts_[dag.get_type()]=23, &dag=0x7f9264d7d250, dag={ObIDag:{this:0x7f9264d7d250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C740-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665670687, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.672062] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C740-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665670684, task_id:YB42C0A8002D-0006230056D8C740-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.672064] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337665635665, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.672072] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=41, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.672083] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.672088] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337665672087, task_id:YB42C0A8002D-0006230056D8C745-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.672092] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=3] schedule one task(task=0x7f9264d7f000, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.672096] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337665672096, task_id:YB42C0A8002D-0006230056D8C746-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.672087] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.672098] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=2] schedule one task(task=0x7f9264d7f530, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.672102] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=12] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:377}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:43, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.672116] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.672119] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.672121] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:05.672125] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.672115] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337665635679, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.672127] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.672119] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:377}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:377}, table_count:1, [{i:0, table_key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665671391, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=709us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.672134] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.672129] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=16, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.672150] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.672139] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C743-0-0, dag_ret:0, dag_status:2, start_time:1727337665671373, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3480, output_block_cnt_arr:0x7f92728f3488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337665671391, estimated_finish_time:1727337682672139}) [2024-09-26 16:01:05.672154] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.672155] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=16] succeed to init merge ctx(task={this:0x7f9264d7e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C743-0-0, dag_ret:0, dag_status:2, start_time:1727337665671373, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672161] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=5] task finish process(ret=0, start_time=1727337665671390, end_time=1727337665672161, runtime=771, *this={this:0x7f9264d7e5a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C743-0-0, dag_ret:0, dag_status:2, start_time:1727337665671373, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672161] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:297}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:18, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.672187] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337665635650, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.672180] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:297}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:297}, table_count:1, [{i:0, table_key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665671408, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=750us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.672195] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752949376, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=20, max_schema_version_in_memtable=1710828752949376, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.672202] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C744-0-0, dag_ret:0, dag_status:2, start_time:1727337665671378, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a43490, output_block_cnt_arr:0x7f9263a43498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337665671408, estimated_finish_time:1727337682672201}) [2024-09-26 16:01:05.672210] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=5] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.672208] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7ead0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C744-0-0, dag_ret:0, dag_status:2, start_time:1727337665671378, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672212] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.672213] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=5] task finish process(ret=0, start_time=1727337665671406, end_time=1727337665672213, runtime=807, *this={this:0x7f9264d7ead0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C744-0-0, dag_ret:0, dag_status:2, start_time:1727337665671378, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672216] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:323}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:22, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752949376, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.672220] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=2] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.672231] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:323}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752949376, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:323}, table_count:1, [{i:0, table_key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665671472, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=743us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.672248] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.672251] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.672253] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:05.672257] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.672257] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C742-0-0, dag_ret:0, dag_status:2, start_time:1727337665671364, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505480, output_block_cnt_arr:0x7f9266505488, concurrent_cnt:1, estimate_row_cnt:205, estimate_occupy_size:0, latest_update_ts:1727337665671472, estimated_finish_time:1727337682672257}) [2024-09-26 16:01:05.672260] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.672263] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d7e070, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C742-0-0, dag_ret:0, dag_status:2, start_time:1727337665671364, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672267] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.672267] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=4] task finish process(ret=0, start_time=1727337665671471, end_time=1727337665672267, runtime=796, *this={this:0x7f9264d7e070, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C742-0-0, dag_ret:0, dag_status:2, start_time:1727337665671364, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672290] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=140, range=0x7f92a9a56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fa060, count:3, allocator:0x7f92636367d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92665fa320, local_datums:0x7f92665fa320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:01:05.672312] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=377, range=0x7f92aa8d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:41, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:43}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d30060, count:5, allocator:0x7f92bc8fc7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:43, datum_buffer:{capacity:43, datums:0x7f9287d312b0, local_datums:0x7f9287d30360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,]}) [2024-09-26 16:01:05.672356] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.672364] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=25][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.672390] ERROR issue_dba_error (ob_log.cpp:1875) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.672393] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.672401] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337665635602, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.672413] ERROR issue_dba_error (ob_log.cpp:1875) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.672416] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.672422] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665672253, end_time=1727337665672422, runtime=169, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73F-0-0, dag_ret:0, dag_status:2, start_time:1727337665670681, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672426] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=0] partition merge iter row count(i=0, row_count=6, ghost_row_count=0, pkey={tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337665635665, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.672432] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1751, dag_cnt=22, dag_cnts_[dag.get_type()]=22, &dag=0x7f9264d7cd20, dag={ObIDag:{this:0x7f9264d7cd20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C73F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665670681, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.672439] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C73F-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665670676, task_id:YB42C0A8002D-0006230056D8C73F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.672447] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665672283, end_time=1727337665672447, runtime=164, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C743-0-0, dag_ret:0, dag_status:2, start_time:1727337665671373, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672456] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1083, dag_cnt=21, dag_cnts_[dag.get_type()]=21, &dag=0x7f9264d7e1e0, dag={ObIDag:{this:0x7f9264d7e1e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C743-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665671373, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.672463] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C743-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665671369, task_id:YB42C0A8002D-0006230056D8C743-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.672520] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=11] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.672525] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=4] schedule one task(task=0x7f92658d4030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.672531] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337665672529, task_id:YB42C0A8002D-0006230056D8C747-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.672534] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=2] schedule one task(task=0x7f9264d7fa60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.672537] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665672537, task_id:YB42C0A8002D-0006230056D8C748-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.672539] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=2] schedule one task(task=0x7f9264d7ff90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.672552] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337665635694, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.672562] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.672574] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.672577] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.672573] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=323, range=0x7f92aa952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:20, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:22}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fa060, count:5, allocator:0x7f92ba7fc7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,]}) [2024-09-26 16:01:05.672593] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101065}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.672612] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:101065}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101065}, table_count:1, [{i:0, table_key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665672111, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=480us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.672636] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C745-0-0, dag_ret:0, dag_status:2, start_time:1727337665672091, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515490, output_block_cnt_arr:0x7f9268515498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337665672111, estimated_finish_time:1727337682672636}) [2024-09-26 16:01:05.672642] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7f000, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C745-0-0, dag_ret:0, dag_status:2, start_time:1727337665672091, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672647] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=5] task finish process(ret=0, start_time=1727337665672110, end_time=1727337665672647, runtime=537, *this={this:0x7f9264d7f000, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C745-0-0, dag_ret:0, dag_status:2, start_time:1727337665672091, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672668] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=2] schedule one task(task=0x7f92658d43c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.672683] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337665635709, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.672692] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.672708] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.672710] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.672711] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.672712] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:05.672714] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.672716] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.672719] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:333}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.672720] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077692196634026, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8653ce0, is_inited:true, wash_score:1727337665635404000, hold_ref_cnt:true, ref_cnt:2, version:2, length:505, tablet_addr:[809012-1445-0-2305843645943099392-2](offset=503808,size=593,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101066}, data_tablet_id:{id:333}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077692196634026, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3243687120, row_count:2118}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8654404, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844488830410752-0](offset=483328,size=1378,type=2,seq=0), type:"MAJOR", tablet_id:{id:101066}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843931558405474-0](offset=484706,size=859,type=2,seq=0), type:"MINI", tablet_id:{id:101066}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337665635723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdd4f0, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221023266}, this:0x7f92a3bdd4f0, timestamp:1727334221023266, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:84, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265274283037001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696471040-2](offset=491520,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843189602812236-0](offset=491852,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879838196-0](offset=492020,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5020, 0x7f92a3bdd4f0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:05.672767] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=46] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.672754] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:333}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:333}, table_count:1, [{i:0, table_key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8651da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665672242, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=475us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.672772] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.672803] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C746-0-0, dag_ret:0, dag_status:2, start_time:1727337665672098, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad490, output_block_cnt_arr:0x7f92684ad498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337665672242, estimated_finish_time:1727337682672803}) [2024-09-26 16:01:05.672810] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7f530, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C746-0-0, dag_ret:0, dag_status:2, start_time:1727337665672098, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672813] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.672815] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=5] task finish process(ret=0, start_time=1727337665672232, end_time=1727337665672815, runtime=583, *this={this:0x7f9264d7f530, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C746-0-0, dag_ret:0, dag_status:2, start_time:1727337665672098, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672847] ERROR issue_dba_error (ob_log.cpp:1875) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.672837] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101065, range=0x7f92aac56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a5e060, count:6, allocator:0x7f92636367d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9264a5e390, local_datums:0x7f9264a5e390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:01:05.672850] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.672859] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=0] partition merge iter row count(i=0, row_count=171, ghost_row_count=0, pkey={tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337665635650, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.672868] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.672877] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665672547, end_time=1727337665672876, runtime=329, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C742-0-0, dag_ret:0, dag_status:2, start_time:1727337665671364, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672885] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1521, dag_cnt=20, dag_cnts_[dag.get_type()]=20, &dag=0x7f9264d7dcb0, dag={ObIDag:{this:0x7f9264d7dcb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C742-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665671364, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.672901] ERROR issue_dba_error (ob_log.cpp:1875) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.672903] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C742-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665671361, task_id:YB42C0A8002D-0006230056D8C742-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.672904] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.672914] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=0] partition merge iter row count(i=0, row_count=15, ghost_row_count=0, pkey={tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337665635694, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.672953] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665672803, end_time=1727337665672953, runtime=150, *this={this:0x7f92658d43c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C745-0-0, dag_ret:0, dag_status:2, start_time:1727337665672091, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.672971] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=880, dag_cnt=19, dag_cnts_[dag.get_type()]=19, &dag=0x7f9264d7ec40, dag={ObIDag:{this:0x7f9264d7ec40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C745-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665672091, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.672978] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C745-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665672087, task_id:YB42C0A8002D-0006230056D8C745-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.672982] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=297, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:16, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:18}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665a6060, count:6, allocator:0x7f92636807d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:18, datum_buffer:{capacity:32, datums:0x7f92665a6390, local_datums:0x7f92665a6390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,]}) [2024-09-26 16:01:05.672995] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=13] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.673000] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337665672999, task_id:YB42C0A8002D-0006230056D8C749-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.673003] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=3] schedule one task(task=0x7f9264d804c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.673007] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665673006, task_id:YB42C0A8002D-0006230056D8C74A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.673007] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.673009] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=2] schedule one task(task=0x7f9264d809f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.673047] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.673050] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.673058] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337665635679, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.673057] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=333, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684e0060, count:7, allocator:0x7f92636327d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f92684e03b0, local_datums:0x7f92684e03b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:01:05.673074] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665672955, end_time=1727337665673074, runtime=119, *this={this:0x7f92658d4030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C744-0-0, dag_ret:0, dag_status:2, start_time:1727337665671378, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.673082] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1704, dag_cnt=18, dag_cnts_[dag.get_type()]=18, &dag=0x7f9264d7e710, dag={ObIDag:{this:0x7f9264d7e710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C744-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665671378, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.673088] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C744-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337665671375, task_id:YB42C0A8002D-0006230056D8C744-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.673116] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.673118] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.673120] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:05.673124] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.673127] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864cf80, is_inited:true, wash_score:1727337665635404000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[814042-5586-0-2305843644870074368-2](offset=1220608,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:378}, data_tablet_id:{id:378}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2166189977, row_count:350}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864d65c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844352465928192-0](offset=1212416,size=1251,type=2,seq=0), type:"MAJOR", tablet_id:{id:378}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063563080818000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843914379265251-0](offset=1213667,size=843,type=2,seq=0), type:"MINI", tablet_id:{id:378}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557779816000, upper_trans_version:1726121557779816000}, {i:1, addr:[814042-5586-0-2305843914379266094-0](offset=1214510,size=843,type=2,seq=0), type:"MINI", tablet_id:{id:378}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986451466000, upper_trans_version:1726149986451466000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337665635767, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdf430, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221743022}, this:0x7f92a3bdf430, timestamp:1727334221743022, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:88, clock:641728512}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265293334630000, v:0}, rec_scn:{val:1726265293334630000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513829376-2](offset=1216512,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843176718635446-0](offset=1216950,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880563282-0](offset=1217106,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447083316-0](offset=1217332,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7f2b0, 0x7f92a3bdf430, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:05.673164] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=37] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.673170] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.673189] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.673191] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.673193] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:05.673196] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.673199] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.673202] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.673228] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.673262] ERROR issue_dba_error (ob_log.cpp:1875) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.673264] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.673274] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=0] partition merge iter row count(i=0, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337665635709, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.673295] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337665673020, end_time=1727337665673295, runtime=275, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C746-0-0, dag_ret:0, dag_status:2, start_time:1727337665672098, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.673303] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1205, dag_cnt=17, dag_cnts_[dag.get_type()]=17, &dag=0x7f9264d7f170, dag={ObIDag:{this:0x7f9264d7f170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C746-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665672098, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.673312] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C746-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337665672096, task_id:YB42C0A8002D-0006230056D8C746-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.673366] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.673369] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.673371] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:05.673375] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.673378] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.673385] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.673404] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665673403, task_id:YB42C0A8002D-0006230056D8C74B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.673408] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=3] schedule one task(task=0x7f9264d80f20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.673412] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337665673412, task_id:YB42C0A8002D-0006230056D8C74C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.673414] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=2] schedule one task(task=0x7f9264d81450, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.673428] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.673431] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.673432] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:05.673435] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.673437] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b861fe60, is_inited:true, wash_score:1727337665635404000, hold_ref_cnt:true, ref_cnt:2, version:2, length:491, tablet_addr:[816017-1849-0-2305843630912028672-2](offset=1818624,size=579,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:215}, data_tablet_id:{id:215}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3095927781, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86204f4, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844118390796288-0](offset=1798144,size=1033,type=2,seq=0), type:"MAJOR", tablet_id:{id:215}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389038459000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843910084883465-0](offset=1799177,size=839,type=2,seq=0), type:"MINOR", tablet_id:{id:215}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726209400113641000, upper_trans_version:1726209400113641000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337665635796, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcf430, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209475373}, this:0x7f92a3bcf430, timestamp:1727334209475373, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:56, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277800266460000, v:0}, rec_scn:{val:1726241800098150000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697789952-2](offset=1810432,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834327372-0](offset=1810764,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881157084-0](offset=1810908,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447677118-0](offset=1811134,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc86d0, 0x7f92a3bcf430, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:05.673477] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=40] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.673483] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.673507] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.673509] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.673511] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:05.673513] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.673515] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.673520] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241548806527000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.673539] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337665635723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.673565] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337665635738, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.673564] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=25] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92665a9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.673578] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.673574] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=19, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.673581] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.673586] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92665a9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101066}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.673611] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.673614] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.673603] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101066}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92665a9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101066}, table_count:1, [{i:0, table_key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8653ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665672703, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=881us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.673619] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:334}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:21, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.673633] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C747-0-0, dag_ret:0, dag_status:2, start_time:1727337665672534, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665a9490, output_block_cnt_arr:0x7f92665a9498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337665672703, estimated_finish_time:1727337682673632}) [2024-09-26 16:01:05.673638] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:334}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:334}, table_count:1, [{i:0, table_key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8654c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665673186, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=431us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.673639] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d7fa60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C747-0-0, dag_ret:0, dag_status:2, start_time:1727337665672534, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.673656] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=17] task finish process(ret=0, start_time=1727337665672702, end_time=1727337665673656, runtime=954, *this={this:0x7f9264d7fa60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C747-0-0, dag_ret:0, dag_status:2, start_time:1727337665672534, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.673706] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337665635796, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.673708] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C748-0-0, dag_ret:0, dag_status:2, start_time:1727337665672539, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d4a0, output_block_cnt_arr:0x7f9264b1d4a8, concurrent_cnt:1, estimate_row_cnt:993, estimate_occupy_size:0, latest_update_ts:1727337665673186, estimated_finish_time:1727337682673708}) [2024-09-26 16:01:05.673716] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d7ff90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C748-0-0, dag_ret:0, dag_status:2, start_time:1727337665672539, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.673715] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752725472, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752725472, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.673721] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=4] task finish process(ret=0, start_time=1727337665673185, end_time=1727337665673720, runtime=535, *this={this:0x7f9264d7ff90, type:15, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C748-0-0, dag_ret:0, dag_status:2, start_time:1727337665672539, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.673724] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.673738] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337665635767, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.673744] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=12] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.673746] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.673754] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.673757] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.673750] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:215}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752725472, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.673764] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:215}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752725472, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:215}, table_count:1, [{i:0, table_key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665673425, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=323us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.673761] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:378}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.673783] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74C-0-0, dag_ret:0, dag_status:2, start_time:1727337665673414, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1460, output_block_cnt_arr:0x7f92728f1468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337665673425, estimated_finish_time:1727337682673783}) [2024-09-26 16:01:05.673789] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d81450, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74C-0-0, dag_ret:0, dag_status:2, start_time:1727337665673414, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.673793] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=4] task finish process(ret=0, start_time=1727337665673424, end_time=1727337665673793, runtime=369, *this={this:0x7f9264d81450, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74C-0-0, dag_ret:0, dag_status:2, start_time:1727337665673414, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.673788] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=23] succeed to build merge ctx(tablet_id={id:378}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:378}, table_count:1, [{i:0, table_key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665673112, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=647us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.673806] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74A-0-0, dag_ret:0, dag_status:2, start_time:1727337665673009, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665a7480, output_block_cnt_arr:0x7f92665a7488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337665673112, estimated_finish_time:1727337682673806}) [2024-09-26 16:01:05.673811] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d809f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74A-0-0, dag_ret:0, dag_status:2, start_time:1727337665673009, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.673811] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337665635753, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.673816] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=5] task finish process(ret=0, start_time=1727337665673110, end_time=1727337665673816, runtime=706, *this={this:0x7f9264d809f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74A-0-0, dag_ret:0, dag_status:2, start_time:1727337665673009, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.673819] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=14, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.673827] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.673829] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.673833] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:451}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:16, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.673845] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=9] succeed to build merge ctx(tablet_id={id:451}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:451}, table_count:1, [{i:0, table_key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8650e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665673361, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=470us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.673859] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=12] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C749-0-0, dag_ret:0, dag_status:2, start_time:1727337665673003, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505470, output_block_cnt_arr:0x7f9266505478, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337665673361, estimated_finish_time:1727337682673859}) [2024-09-26 16:01:05.673864] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d804c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C749-0-0, dag_ret:0, dag_status:2, start_time:1727337665673003, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.673868] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=4] task finish process(ret=0, start_time=1727337665673360, end_time=1727337665673868, runtime=508, *this={this:0x7f9264d804c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C749-0-0, dag_ret:0, dag_status:2, start_time:1727337665673003, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.673882] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752511928, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.673890] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.673892] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.673904] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=10] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:121}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752511928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.673918] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:121}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665673504, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=398us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.673933] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74B-0-0, dag_ret:0, dag_status:2, start_time:1727337665673407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f480, output_block_cnt_arr:0x7f9264b1f488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337665673504, estimated_finish_time:1727337682673933}) [2024-09-26 16:01:05.673938] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d80f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74B-0-0, dag_ret:0, dag_status:2, start_time:1727337665673407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:05.673942] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=4] task finish process(ret=0, start_time=1727337665673503, end_time=1727337665673942, runtime=439, *this={this:0x7f9264d80f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74B-0-0, dag_ret:0, dag_status:2, start_time:1727337665673407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:05.673954] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=2] schedule one task(task=0x7f926644c030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.673958] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=3] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.673963] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=2] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.673966] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=2] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.673968] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=1] schedule one task(task=0x7f9264c803c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.673972] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=2] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.674009] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101066, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:7}, cols_desc:{ObIArray:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665b0060, count:7, allocator:0x7f92691a27d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f92665b03b0, local_datums:0x7f92665b03b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:01:05.674141] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.674191] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.674194] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.674201] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=0] partition merge iter row count(i=0, row_count=182, ghost_row_count=0, pkey={tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337665635723, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.674220] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665673982, end_time=1727337665674220, runtime=238, *this={this:0x7f926644c030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C747-0-0, dag_ret:0, dag_status:2, start_time:1727337665672534, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.674229] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1695, dag_cnt=16, dag_cnts_[dag.get_type()]=16, &dag=0x7f9264d7f6a0, dag={ObIDag:{this:0x7f9264d7f6a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C747-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665672534, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.674235] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C747-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665672529, task_id:YB42C0A8002D-0006230056D8C747-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.674288] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=334, range=0x7f92a98d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:19, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:21}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d2e060, count:8, allocator:0x7f92a31fe7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:21, datum_buffer:{capacity:32, datums:0x7f9287d2e3e0, local_datums:0x7f9287d2e3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,]}) [2024-09-26 16:01:05.674498] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.674528] ERROR issue_dba_error (ob_log.cpp:1875) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.674531] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.674539] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=0] partition merge iter row count(i=0, row_count=153, ghost_row_count=0, pkey={tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337665635738, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.674557] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665674262, end_time=1727337665674557, runtime=295, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C748-0-0, dag_ret:0, dag_status:2, start_time:1727337665672539, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.674565] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2026, dag_cnt=15, dag_cnts_[dag.get_type()]=15, &dag=0x7f9264d7fbd0, dag={ObIDag:{this:0x7f9264d7fbd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C748-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665672539, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.674572] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C748-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337665672537, task_id:YB42C0A8002D-0006230056D8C748-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.674615] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=451, range=0x7f92a9cd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:14, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:16}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fa060, count:4, allocator:0x7f92a31fe7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:16, datum_buffer:{capacity:32, datums:0x7f92665fa350, local_datums:0x7f92665fa350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,]}) [2024-09-26 16:01:05.674638] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.674665] ERROR issue_dba_error (ob_log.cpp:1875) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.674667] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.674675] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337665635753, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.674690] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665674592, end_time=1727337665674690, runtime=98, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C749-0-0, dag_ret:0, dag_status:2, start_time:1727337665673003, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.674697] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1694, dag_cnt=14, dag_cnts_[dag.get_type()]=14, &dag=0x7f9264d80100, dag={ObIDag:{this:0x7f9264d80100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C749-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665673003, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.674703] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C749-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665672999, task_id:YB42C0A8002D-0006230056D8C749-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.674742] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=378, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:5, allocator:0x7f92865a27d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:01:05.674785] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.674787] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.674794] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=1] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337665635767, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.674807] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665674722, end_time=1727337665674807, runtime=85, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74A-0-0, dag_ret:0, dag_status:2, start_time:1727337665673009, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.674815] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1806, dag_cnt=13, dag_cnts_[dag.get_type()]=13, &dag=0x7f9264d80630, dag={ObIDag:{this:0x7f9264d80630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665673009, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.674821] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74A-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665673006, task_id:YB42C0A8002D-0006230056D8C74A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.674862] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=121, range=0x7f92a9b52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d34060, count:5, allocator:0x7f92b32387d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9287d34360, local_datums:0x7f9287d34360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:01:05.674902] INFO [STORAGE] dump_info (ob_sstable_struct.cpp:262) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=10] dump merge info(msg="macro block builder close", output_row_per_s=0, new_macro_KB_per_s=0, *this={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:61, merge_start_time:1727337665674840, merge_finish_time:1727337665674901, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:0, max:0, sum:0, count:0}}, {type:"cost_time", info:{min:0, max:0, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:0, max:0, sum:0, count:0}}, {type:"incremental_row_count", info:{min:0, max:0, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:0, snapshot_version:0, start_scn:0, end_scn:0}, macro_id_list:"", comment:""}) [2024-09-26 16:01:05.674918] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=15] partition merge iter row count(i=0, row_count=0, ghost_row_count=0, pkey={tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, table={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337665635782, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.674929] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1544) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=9] merge macro blocks ok(idx_=0, task={this:0x7f9264c803c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74B-0-0, dag_ret:0, dag_status:2, start_time:1727337665673407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:05.674939] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=6] task finish process(ret=0, start_time=1727337665674838, end_time=1727337665674939, runtime=101, *this={this:0x7f9264c803c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74B-0-0, dag_ret:0, dag_status:2, start_time:1727337665673407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:05.674980] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=215, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:6}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d34060, count:3, allocator:0x7f92636327d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9287d34320, local_datums:0x7f9287d34320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:01:05.675001] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=9][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.675030] ERROR issue_dba_error (ob_log.cpp:1875) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.675032] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.675040] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=0] partition merge iter row count(i=0, row_count=9, ghost_row_count=0, pkey={tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337665635796, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.675053] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665674959, end_time=1727337665675053, runtime=94, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74C-0-0, dag_ret:0, dag_status:2, start_time:1727337665673414, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.675060] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1646, dag_cnt=12, dag_cnts_[dag.get_type()]=12, &dag=0x7f9264d81090, dag={ObIDag:{this:0x7f9264d81090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665673414, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.675066] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74C-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337665673412, task_id:YB42C0A8002D-0006230056D8C74C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.676907] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=2] schedule one task(task=0x7f9264c805e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.676916] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] succeed to add sys task(task={start_time:1727337665676914, task_id:YB42C0A8002D-0006230056D8C74D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.676921] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=5] schedule one task(task=0x7f9264d81980, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.676925] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665676924, task_id:YB42C0A8002D-0006230056D8C74E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.676927] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=2] schedule one task(task=0x7f9264d81eb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.676930] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665676930, task_id:YB42C0A8002D-0006230056D8C74F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.676932] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=2] schedule one task(task=0x7f9264d823e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.676936] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665676935, task_id:YB42C0A8002D-0006230056D8C750-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.676938] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=2] schedule one task(task=0x7f9264d82910, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.676943] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337665676942, task_id:YB42C0A8002D-0006230056D8C751-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.676945] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=2] schedule one task(task=0x7f9264d82e40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.676953] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.676959] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.676962] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:05.676964] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=20][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.676966] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.676968] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.676970] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:05.676970] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.676976] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.676977] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.676979] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.676986] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.677008] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.677011] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.677012] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:05.677015] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.677027] INFO [STORAGE.COMPACTION] get_merged_sstable (ob_tablet_merge_task.cpp:1285) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=5] create new merged sstable(ctx.param_.tablet_id_={id:121}, snapshot_version=1726241132764173999, ctx.param_.merge_type_=3, ctx.create_snapshot_version_=0, table_mode_flag=0) [2024-09-26 16:01:05.677035] INFO [STORAGE] close (ob_index_block_builder.cpp:815) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=4] succeed to close sstable index builder(res={root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_block_ids_.count():0, other_block_ids_.count():0, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, data_column_cnt:10, data_column_checksums:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], row_count:0, max_merged_trans_version:0, contain_uncommitted_row:false, occupy_size:0, original_size:0, data_checksum:0, use_old_macro_block_count:0, compressor_type:1, root_row_store_type:0, nested_offset:0, nested_size:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}) [2024-09-26 16:01:05.677018] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b864fe60, is_inited:true, wash_score:1727337665635404000, hold_ref_cnt:true, ref_cnt:2, version:2, length:510, tablet_addr:[816017-1849-0-2305843651313397760-2](offset=2093056,size=598,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101063}, data_tablet_id:{id:297}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726063409221604000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1864622635, row_count:24}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86505a8, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844568288894976-0](offset=2072576,size=1452,type=2,seq=0), type:"MAJOR", tablet_id:{id:101063}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063202518238000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843928338769324-0](offset=2074028,size=856,type=2,seq=0), type:"MINI", tablet_id:{id:101063}, scn_range:{start_scn:{val:1726063409221604000, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557514739000, upper_trans_version:1726121557514739000}, {i:1, addr:[816017-1849-0-2305843928338770180-0](offset=2074884,size=856,type=2,seq=0), type:"MINI", tablet_id:{id:101063}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149603136610000, upper_trans_version:1726149603136610000}, {i:2, addr:[816017-1849-0-2305843927265029212-0](offset=2075740,size=855,type=2,seq=0), type:"MINI", tablet_id:{id:101063}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726207962418314000, upper_trans_version:1726207962418314000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337665635881, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd8ea0, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220519573}, this:0x7f92a3bd8ea0, timestamp:1727334220519573, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:75, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726264804632655000, v:0}, rec_scn:{val:1726264804632655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843593331331072-2](offset=2084864,size=544,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046856736-0](offset=2085408,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881431758-0](offset=2085582,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bca610, 0x7f92a3bd8ea0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:05.677060] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=42] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.677065] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.677048] INFO [STORAGE] init (ob_sstable.cpp:118) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=8] succeeded to init sstable(ret=0, this={ObITable:{this:0x7f9264b1e128, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9264b1e128, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:null, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:01:05.677088] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=7][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.677091] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.677093] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:05.677096] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.677098] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.677102] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.677068] INFO [STORAGE.COMPACTION] create_sstable (ob_tablet_merge_ctx.cpp:367) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=19] succeed to merge sstable(param={table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_logic_seq:0, schema_version:1710828752511928, create_snapshot_version:0, progressive_merge_round:0, progressive_merge_step:0, is_ready_for_read:true, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, index_type:0, root_block_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), root_block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, root_row_store_type:0, latest_row_store_type:1, data_index_tree_height:0, data_block_macro_meta_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), data_block_macro_meta:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, use_old_macro_block_count:0, row_count:0, rowkey_column_cnt:5, column_cnt:10, column_checksums:[], data_checksum:0, occupy_size:0, original_size:0, max_merged_trans_version:1726241132764173999, ddl_scn:{val:0, v:0}, contain_uncommitted_row:false, is_meta_root:false, compressor_type:1, encrypt_id:0, master_key_id:0, recycle_version:0, nested_offset:0, nested_size:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}, table_key={tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_merge_info={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:1397, merge_start_time:1727337665673504, merge_finish_time:1727337665674901, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"cost_time", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"incremental_row_count", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:1, snapshot_version:0, start_scn:1726211867305630000, end_scn:1726241548806527000}, macro_id_list:"", comment:""}) [2024-09-26 16:01:05.677185] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.677188] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.677190] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:05.677193] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.677196] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8623ce0, is_inited:true, wash_score:1727337665635404000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[816017-1849-0-2305843644870885376-2](offset=2031616,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:154}, data_tablet_id:{id:154}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726074677203981000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3786676695, row_count:578}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86243bc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844492053168128-0](offset=2015232,size=1381,type=2,seq=0), type:"MAJOR", tablet_id:{id:154}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726074015354550000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843989541995877-0](offset=2016613,size=913,type=2,seq=0), type:"MINOR", tablet_id:{id:154}, scn_range:{start_scn:{val:1726074677203981000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726205802467242000, upper_trans_version:1726205802467242000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337665635836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd5020, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209508489}, this:0x7f92a3bd5020, timestamp:1727334209508489, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:67, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277803107894000, v:0}, rec_scn:{val:1726241804194470000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843367845486592-2](offset=2023424,size=334,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719442254-0](offset=2023758,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881370090-0](offset=2023914,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447890124-0](offset=2024140,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc6f60, 0x7f92a3bd5020, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:05.677236] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=40] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.677242] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.677357] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337665635819, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.677358] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=21] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337665635851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.677368] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753355744, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753355744, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.677382] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.677393] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.677388] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=29] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.677401] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:456}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753355744, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.677402] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.677434] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=31] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.677438] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=25] succeed to build merge ctx(tablet_id={id:456}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753355744, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:456}, table_count:1, [{i:0, table_key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8634c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665676937, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=461us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.677486] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=41] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74D-0-0, dag_ret:0, dag_status:2, start_time:1727337665676921, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a43490, output_block_cnt_arr:0x7f9263a43498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337665676937, estimated_finish_time:1727337682677486}) [2024-09-26 16:01:05.677492] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d81980, type:15, status:2, dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74D-0-0, dag_ret:0, dag_status:2, start_time:1727337665676921, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677486] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101015}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.677497] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=5] task finish process(ret=0, start_time=1727337665676934, end_time=1727337665677497, runtime=563, *this={this:0x7f9264d81980, type:15, status:2, dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74D-0-0, dag_ret:0, dag_status:2, start_time:1727337665676921, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677523] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337665635881, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.677514] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:101015}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9264b1d330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101015}, table_count:1, [{i:0, table_key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665676959, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=524us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.677531] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.677541] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.677543] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.677541] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74F-0-0, dag_ret:0, dag_status:2, start_time:1727337665676932, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1d490, output_block_cnt_arr:0x7f9264b1d498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337665676959, estimated_finish_time:1727337682677541}) [2024-09-26 16:01:05.677548] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d823e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74F-0-0, dag_ret:0, dag_status:2, start_time:1727337665676932, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677553] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=5] task finish process(ret=0, start_time=1727337665676957, end_time=1727337665677553, runtime=596, *this={this:0x7f9264d823e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74F-0-0, dag_ret:0, dag_status:2, start_time:1727337665676932, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677548] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101063}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.677564] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101063}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9263a45330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101063}, table_count:1, [{i:0, table_key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665677004, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=542us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.677582] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C751-0-0, dag_ret:0, dag_status:2, start_time:1727337665676945, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a454a0, output_block_cnt_arr:0x7f9263a454a8, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337665677004, estimated_finish_time:1727337682677581}) [2024-09-26 16:01:05.677587] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=4] succeed to init merge ctx(task={this:0x7f9264d82e40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C751-0-0, dag_ret:0, dag_status:2, start_time:1727337665676945, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677591] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=4] task finish process(ret=0, start_time=1727337665677002, end_time=1727337665677591, runtime=589, *this={this:0x7f9264d82e40, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C751-0-0, dag_ret:0, dag_status:2, start_time:1727337665676945, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677599] INFO [STORAGE] build_new_table_store (ob_tablet_table_store.cpp:1206) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=36] succeed to build new table store(major_tables_={cnt:1, is_inited:true}, minor_tables_={cnt:3, is_inited:true}, memtables_={count:1}, PRINT_TS(*this)={ObTabletTableStore_Pretty:this:0x7f92a9ed3ca8, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, memtables:{count:1}, is_ready_for_read:true, table_array:{ [2877] [ YB42C0A8002D-0006230056D8C74B-0-0 ] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [2877] [ YB42C0A8002D-0006230056D8C74B-0-0 ] MAJOR MAJOR 1725120003420849000 1725120003420849000 {val:0, v:0} {val:1726077600415496000, v:0} 0 false [2877] [ YB42C0A8002D-0006230056D8C74B-0-0 ] MINOR MINOR 1726178585269925002 1726178585269925002 {val:1726077259159067000, v:0} {val:1726178585269925002, v:0} 0 false [2877] [ YB42C0A8002D-0006230056D8C74B-0-0 ] MINI 1726211867106157000 1726211867106157000 {val:1726178585269925002, v:0} {val:1726211867305630000, v:0} 0 false [2877] [ YB42C0A8002D-0006230056D8C74B-0-0 ] MINI 1726241132764173999 1726241132764173999 {val:1726211867305630000, v:0} {val:1726241548806527000, v:0} 0 false [2877] [ YB42C0A8002D-0006230056D8C74B-0-0 ] MEM ACTIVE 9223372036854775807 18446744073709547609 {val:1726241548806527000, v:0} {val:4611686018427387903, v:0} 3 unused }}) [2024-09-26 16:01:05.677606] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337665635865, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.677613] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f9287d2f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.677621] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.677623] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.677626] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9287d2f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101016}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.677637] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337665635836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.677640] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:101016}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f9287d2f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101016}, table_count:1, [{i:0, table_key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665677085, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=540us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.677647] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.677656] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.677657] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=12] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C750-0-0, dag_ret:0, dag_status:2, start_time:1727337665676938, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287d2f490, output_block_cnt_arr:0x7f9287d2f498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337665677085, estimated_finish_time:1727337682677657}) [2024-09-26 16:01:05.677659] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.677661] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=4] succeed to init merge ctx(task={this:0x7f9264d82910, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C750-0-0, dag_ret:0, dag_status:2, start_time:1727337665676938, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677666] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=5] task finish process(ret=0, start_time=1727337665677084, end_time=1727337665677666, runtime=582, *this={this:0x7f9264d82910, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C750-0-0, dag_ret:0, dag_status:2, start_time:1727337665676938, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677663] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:154}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.677680] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:154}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:154}, table_count:1, [{i:0, table_key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8623ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665677181, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=481us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.677699] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74E-0-0, dag_ret:0, dag_status:2, start_time:1727337665676927, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1480, output_block_cnt_arr:0x7f92728f1488, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337665677181, estimated_finish_time:1727337682677699}) [2024-09-26 16:01:05.677705] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d81eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74E-0-0, dag_ret:0, dag_status:2, start_time:1727337665676927, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677709] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=4] task finish process(ret=0, start_time=1727337665677179, end_time=1727337665677709, runtime=530, *this={this:0x7f9264d81eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74E-0-0, dag_ret:0, dag_status:2, start_time:1727337665676927, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677722] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=2] schedule one task(task=0x7f9264d84ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.677728] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=3] schedule one task(task=0x7f92658d4030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.677731] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=2] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.677735] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=2] schedule one task(task=0x7f9264d85260, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.677738] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=2] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.677784] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=154, range=0x7f92a9d52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92729f0060, count:5, allocator:0x7f9269ffe7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f92729f0360, local_datums:0x7f92729f0360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:01:05.677791] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101016, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:6, allocator:0x7f92636907d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:01:05.677806] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=456, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92729fa060, count:7, allocator:0x7f92636927d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f92729fa3b0, local_datums:0x7f92729fa3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:01:05.677852] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=23][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.677863] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.677893] ERROR issue_dba_error (ob_log.cpp:1875) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.677896] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.677906] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337665635865, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.677918] ERROR issue_dba_error (ob_log.cpp:1875) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.677921] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.677925] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.677929] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.677930] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665677763, end_time=1727337665677930, runtime=167, *this={this:0x7f9264d85260, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C750-0-0, dag_ret:0, dag_status:2, start_time:1727337665676938, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677930] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337665635836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.677940] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=1] dag finished(dag_ret=-4184, runtime=1002, dag_cnt=11, dag_cnts_[dag.get_type()]=11, &dag=0x7f9264d82550, dag={ObIDag:{this:0x7f9264d82550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C750-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665676938, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.677948] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C750-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337665676935, task_id:YB42C0A8002D-0006230056D8C750-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.677939] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=0] partition merge iter row count(i=0, row_count=45, ghost_row_count=0, pkey={tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337665635819, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.677949] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665677746, end_time=1727337665677949, runtime=203, *this={this:0x7f92658d4030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74E-0-0, dag_ret:0, dag_status:2, start_time:1727337665676927, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677964] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665677765, end_time=1727337665677963, runtime=198, *this={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74D-0-0, dag_ret:0, dag_status:2, start_time:1727337665676921, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.677975] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1053, dag_cnt=10, dag_cnts_[dag.get_type()]=10, &dag=0x7f9264d815c0, dag={ObIDag:{this:0x7f9264d815c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665676921, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.677983] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74D-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665676914, task_id:YB42C0A8002D-0006230056D8C74D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.678008] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665678004, task_id:YB42C0A8002D-0006230056D8C752-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.678011] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=3] schedule one task(task=0x7f9264d83370, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.678018] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337665678017, task_id:YB42C0A8002D-0006230056D8C753-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.678020] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=2] schedule one task(task=0x7f9264d838a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.678024] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101015, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:6, allocator:0x7f92864e07d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9264a62390, local_datums:0x7f9264a62390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:01:05.678035] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.678038] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.678040] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:05.678046] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.678058] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101063, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:8, allocator:0x7f92636327d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9263b5e3e0, local_datums:0x7f9263b5e3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:05.678049] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.678078] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=25] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.678078] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.678082] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.678108] ERROR issue_dba_error (ob_log.cpp:1875) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.678110] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.678110] INFO [MDS] init_for_evict_medium_info (ob_tablet_mds_data.cpp:414) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=14] succeeded to init mds data(ret=0, finish_medium_scn=1726077600415496000, src_medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, src_extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}) [2024-09-26 16:01:05.678114] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.678117] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.678118] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337665635851, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.678125] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=0] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337665635881, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.678136] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665677982, end_time=1727337665678135, runtime=153, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74F-0-0, dag_ret:0, dag_status:2, start_time:1727337665676932, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.678144] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1212, dag_cnt=9, dag_cnts_[dag.get_type()]=9, &dag=0x7f9264d82020, dag={ObIDag:{this:0x7f9264d82020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665676932, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.678144] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665678012, end_time=1727337665678144, runtime=132, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C751-0-0, dag_ret:0, dag_status:2, start_time:1727337665676945, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.678152] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74F-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665676930, task_id:YB42C0A8002D-0006230056D8C74F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.678176] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.678179] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.678181] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:05.678186] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.678187] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1260, dag_cnt=8, dag_cnts_[dag.get_type()]=8, &dag=0x7f9264d81af0, dag={ObIDag:{this:0x7f9264d81af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665676927, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.678193] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74E-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665676924, task_id:YB42C0A8002D-0006230056D8C74E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.678126] INFO [STORAGE] init_for_merge (ob_tablet.cpp:363) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=7] succeeded to init tablet for mini/minor/major merge(ret=0, param={sstable:0x7f9264b1e128, snapshot_version:1726241132764173999, clog_checkpoint_scn:{val:1726241548806527000, v:0}, multi_version_start:1726178585269925002, need_report:false, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, rebuild_seq:0, update_with_major_flag:false, need_check_sstable:false, ddl_info:{keep_old_ddl_sstable:true, ddl_start_scn:{val:0, v:0}, ddl_commit_scn:{val:0, v:0}, ddl_checkpoint_scn:{val:0, v:0}, ddl_snapshot_version:0, ddl_execution_id:-1, data_format_version:0}, allow_duplicate_sstable:false, merge_type:"MINI_MERGE", need_check_transfer_seq:true, transfer_seq:0}, old_tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337665635404000, hold_ref_cnt:true, ref_cnt:3, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337665635782, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}, this={this:0x7f9264b20060, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x7f92a9ed4450, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726241548806527000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726241132764173999, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f9264b20670, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}, {i:2, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726241132764173999, upper_trans_version:1726241132764173999}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x7f9264b20520, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9264b21880, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9264b21950, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{redefined:false, snapshot_version:0, schema_version:0, data_tablet_id:{id:0}, hidden_tablet_id:{id:0}, lob_meta_tablet_id:{id:50121}, lob_piece_tablet_id:{id:60121}, is_old_mds:true}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:01:05.678189] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b8652d40, is_inited:true, wash_score:1727337665635404000, hold_ref_cnt:true, ref_cnt:2, version:2, length:504, tablet_addr:[816017-1849-0-2305843644870942720-2](offset=2088960,size=592,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:452}, data_tablet_id:{id:452}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2139885871, row_count:290}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b865341c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844508159356928-0](offset=2076672,size=1396,type=2,seq=0), type:"MAJOR", tablet_id:{id:452}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562830808000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843992763282804-0](offset=2078068,size=916,type=2,seq=0), type:"MINI", tablet_id:{id:452}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986149768000, upper_trans_version:1726149986149768000}, {i:1, addr:[816017-1849-0-2305844002426960136-0](offset=2078984,size=925,type=2,seq=0), type:"MINI", tablet_id:{id:452}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726208018712107000, upper_trans_version:1726208018712107000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337665635895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd9670, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220578716}, this:0x7f92a3bd9670, timestamp:1727334220578716, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:76, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265292959510000, v:0}, rec_scn:{val:1726264902486897000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843480588435456-2](offset=2080768,size=439,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719499703-0](offset=2081207,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881427539-0](offset=2081363,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447947573-0](offset=2081589,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc9e40, 0x7f92a3bd9670, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:05.678229] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=40] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.678236] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.678251] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1306, dag_cnt=7, dag_cnts_[dag.get_type()]=7, &dag=0x7f9264d82a80, dag={ObIDag:{this:0x7f9264d82a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C751-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665676945, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.678260] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C751-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337665676942, task_id:YB42C0A8002D-0006230056D8C751-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.678231] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=92] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287d2e060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}}, ref_cnt:0, upper_trans_version:1725120003420849000, timestamp:0}, this:0x7f9287d2e060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1725120003420849000, max_merged_trans_version:1725120003420849000, data_macro_block_count:1, nested_size:4096, nested_offset:589824, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:84, row_count_:14, occupy_size_:1604, original_size_:1914, data_checksum_:3587602083, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:1, data_micro_block_count_:1, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:2, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1725120003420849000, max_merged_trans_version_:1725120003420849000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287d2e310, column_checksum_count:10, data_root_info:{addr:[0-0-0-3458764933653594112-0](offset=0,size=391,type=3,seq=0), block_data:{buf:0x7f9287d2e360, size:391, extra_buf:0x7f9287d2e4e7, extra_size:424, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-3458765009889263616-0](offset=0,size=462,type=3,seq=0), block_data:{buf:0x7f9287d2e68f, size:462, extra_buf:null, extra_size:0, type:1}}, data_block_count:1, other_block_count:0, linked_block_count:0, data_block_ids:0x7f9287d2e85d, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:true, nested_offset:589824, nested_size:4096}}, valid_for_reading:true}) [2024-09-26 16:01:05.678267] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=33] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9264b22060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}}, ref_cnt:0, upper_trans_version:1726178585269925002, timestamp:0}, this:0x7f9264b22060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726178585269925002, max_merged_trans_version:1726178585269925002, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726178585269925002, v:0}, meta:{basic_meta:{version:1, length:94, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726178585269925002, max_merged_trans_version_:1726178585269925002, recycle_version:1726077600415496000, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726178585269925002, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:1, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9264b22310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:01:05.678279] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=11] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9287d2e060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}}, ref_cnt:0, upper_trans_version:1726211867106157000, timestamp:0}, this:0x7f9287d2e060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726211867106157000, max_merged_trans_version:1726211867106157000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726211867305630000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726211867106157000, max_merged_trans_version_:1726211867106157000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726211867305630000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9287d2e310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:01:05.678298] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=17] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9264b22060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9264b22060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9264b22310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:01:05.678308] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=9][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.678342] ERROR issue_dba_error (ob_log.cpp:1875) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.678344] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf9de5b1 0xf9dcfd6 0xe51c6fb 0xe519c03 0xe518f89 0xe516f8c 0xf01158d 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.678362] INFO [STORAGE] reset (ob_tablet.cpp:159) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=0] reset tablet(this=0x7f9264b20060, ls_id={id:1}, tablet_id={id:121}, lbt()="0x121f1f9c 0xe52cc09 0xe52c6c5 0xe52bd13 0x4fd348f 0xf01110c 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:05.678370] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1250) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=0][errcode=-4184] sstable merge finish(ret=-4184, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74B-0-0, dag_ret:0, dag_status:2, start_time:1727337665673407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f480, output_block_cnt_arr:0x7f9264b1f488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337665673504, estimated_finish_time:1727337682673933}, compaction_filter:NULL, time_guard:total=3.60ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, task={this:0x7f9264c805e0, type:16, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74B-0-0, dag_ret:0, dag_status:2, start_time:1727337665673407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:05.678396] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665677025, end_time=1727337665678396, runtime=1371, *this={this:0x7f9264c805e0, type:16, status:2, dag:{ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74B-0-0, dag_ret:0, dag_status:2, start_time:1727337665673407, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:05.678404] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4996, dag_cnt=6, dag_cnts_[dag.get_type()]=6, &dag=0x7f9264d80b60, dag={ObIDag:{this:0x7f9264d80b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C74B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665673407, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}) [2024-09-26 16:01:05.678410] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C74B-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337665673403, task_id:YB42C0A8002D-0006230056D8C74B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.678433] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337665635910, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.678442] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=8] update column cnt(column_cnt_=26, store_column_cnt_=26, column_cnt_=26, column_array_.count()=0) [2024-09-26 16:01:05.678444] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92728f7480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=26, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.678456] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.678459] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.678485] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=23] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f7480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:328}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:28, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.678503] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:328}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92728f7480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:328}, table_count:1, [{i:0, table_key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e072bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665678030, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=453us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.678523] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d834e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C753-0-0, dag_ret:0, dag_status:2, start_time:1727337665678020, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f75d0, output_block_cnt_arr:0x7f92728f75d8, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337665678030, estimated_finish_time:1727337682678522}) [2024-09-26 16:01:05.678528] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d838a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d834e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C753-0-0, dag_ret:0, dag_status:2, start_time:1727337665678020, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.678533] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=4] task finish process(ret=0, start_time=1727337665678029, end_time=1727337665678533, runtime=504, *this={this:0x7f9264d838a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d834e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C753-0-0, dag_ret:0, dag_status:2, start_time:1727337665678020, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.678565] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=2] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.678571] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337665678569, task_id:YB42C0A8002D-0006230056D8C754-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.678574] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=3] schedule one task(task=0x7f9264d83dd0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.678578] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665678577, task_id:YB42C0A8002D-0006230056D8C755-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.678580] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=2] schedule one task(task=0x7f9264d84300, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.678585] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337665678584, task_id:YB42C0A8002D-0006230056D8C756-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.678587] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=2] schedule one task(task=0x7f9264d84830, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.678590] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337665678590, task_id:YB42C0A8002D-0006230056D8C757-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.678592] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=2] schedule one task(task=0x7f9264d84d60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.678607] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.678610] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.678612] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:05.678615] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.678618] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.678624] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.678616] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=328, range=0x7f92aab52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:26, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:28}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a5e060, count:5, allocator:0x7f92b33d87d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:28, datum_buffer:{capacity:32, datums:0x7f9264a5e360, local_datums:0x7f9264a5e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,]}) [2024-09-26 16:01:05.678640] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.678642] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.678643] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:05.678646] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.678648] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.678652] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.678697] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.678699] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.678701] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:05.678704] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.678706] ERROR issue_dba_error (ob_log.cpp:1875) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.678709] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.678718] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337665635910, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.678706] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077706539141067, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f926e107fe0, is_inited:true, wash_score:1727337665635404000, hold_ref_cnt:true, ref_cnt:2, version:2, length:516, tablet_addr:[809012-1445-0-2305843657754198016-2](offset=442368,size=604,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:330}, data_tablet_id:{id:330}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077706539141067, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3803410633, row_count:15330}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f926e108704, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844604794474496-0](offset=430080,size=1486,type=2,seq=0), type:"MAJOR", tablet_id:{id:330}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562805980000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843948738221518-0](offset=431566,size=875,type=2,seq=0), type:"MINI", tablet_id:{id:330}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986144852000, upper_trans_version:1726149986144852000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337665635924, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdade0, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220705914}, this:0x7f92a3bdade0, timestamp:1727334220705914, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:79, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265269081598001, v:0}, rec_scn:{val:1726264907253071001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696413696-2](offset=434176,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843189602754892-0](offset=434508,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879780852-0](offset=434676,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843216446300886-0](offset=434902,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc9670, 0x7f92a3bdade0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:05.678741] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=35] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.678737] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665678578, end_time=1727337665678737, runtime=159, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d834e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C753-0-0, dag_ret:0, dag_status:2, start_time:1727337665678020, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.678745] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.678759] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=728, dag_cnt=5, dag_cnts_[dag.get_type()]=5, &dag=0x7f9264d834e0, dag={ObIDag:{this:0x7f9264d834e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C753-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665678020, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.678768] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C753-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337665678017, task_id:YB42C0A8002D-0006230056D8C753-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.678805] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.678808] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:05.678810] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:05.678814] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:05.678817] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:05.678832] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=11] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.678923] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337665635953, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.678935] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=25, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.678946] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.678949] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.678954] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:332}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:27, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.678971] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:332}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:332}, table_count:1, [{i:0, table_key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8648b00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665678603, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=349us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.679012] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=35] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d84470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C756-0-0, dag_ret:0, dag_status:2, start_time:1727337665678587, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728fb490, output_block_cnt_arr:0x7f92728fb498, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337665678603, estimated_finish_time:1727337682679011}) [2024-09-26 16:01:05.679018] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d84830, type:15, status:2, dag:{ObIDag:{this:0x7f9264d84470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C756-0-0, dag_ret:0, dag_status:2, start_time:1727337665678587, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.679023] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=4] task finish process(ret=0, start_time=1727337665678601, end_time=1727337665679022, runtime=421, *this={this:0x7f9264d84830, type:15, status:2, dag:{ObIDag:{this:0x7f9264d84470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C756-0-0, dag_ret:0, dag_status:2, start_time:1727337665678587, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.679052] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337665635924, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.679061] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=8] update column cnt(column_cnt_=12, store_column_cnt_=12, column_cnt_=12, column_array_.count()=0) [2024-09-26 16:01:05.679064] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9266505490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.679080] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.679082] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.679087] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:330}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.679102] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:330}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9266505490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:330}, table_count:1, [{i:0, table_key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e107fe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665678693, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=392us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.679125] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C754-0-0, dag_ret:0, dag_status:2, start_time:1727337665678574, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665055f0, output_block_cnt_arr:0x7f92665055f8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337665678693, estimated_finish_time:1727337682679125}) [2024-09-26 16:01:05.679130] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264d83dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C754-0-0, dag_ret:0, dag_status:2, start_time:1727337665678574, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.679135] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=5] task finish process(ret=0, start_time=1727337665678692, end_time=1727337665679135, runtime=443, *this={this:0x7f9264d83dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C754-0-0, dag_ret:0, dag_status:2, start_time:1727337665678574, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.679146] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=2] schedule one task(task=0x7f9264d84ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.679150] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=3] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.679193] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=330, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5e060, count:7, allocator:0x7f92a31fe7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9264d5e3b0, local_datums:0x7f9264d5e3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:01:05.679204] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=332, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:25, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:27}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d62060, count:6, allocator:0x7f92636807d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:27, datum_buffer:{capacity:32, datums:0x7f9264d62390, local_datums:0x7f9264d62390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,]}) [2024-09-26 16:01:05.679239] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:05.679273] ERROR issue_dba_error (ob_log.cpp:1875) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.679275] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.679282] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:05.679284] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:05.679285] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be4070, hex: 0000000000000000, int: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be40a8, hex: DCA4070000000000, int: 500956},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be40e0, hex: DCA4070000000000, int: 500956},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be41b0, hex: 647770AD00220600, int: 1726236165437284},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be4150, hex: B078BA4A572D0BE8, int: -1726236179227314000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9263be4188, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:8528, column_count:27, rowkey_column_count:6, has_column_checksum:0, row_count:5, row_store_type:0, opt:5, var_column_count:0, row_offset:845, original_length:805, max_merged_trans_version:1726236510704198000, data_length:805, data_zlength:805, data_checksum:2523066956, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f926f804090, buf_size:805, data_size:805, row_count:5, column_count:27, max_merged_trans_version:1726236510704198000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1298280702, row_count_delta:5, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:805}) [2024-09-26 16:01:05.679297] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=11][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:05.679298] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=1][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:01:05.679300] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=2][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:01:05.679301] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=1][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:01:05.679303] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=2][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:01:05.679315] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=12] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337665635953, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.679328] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=10][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:05.679335] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:05.679338] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:05.679339] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337665679174, end_time=1727337665679339, runtime=165, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d84470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C756-0-0, dag_ret:0, dag_status:2, start_time:1727337665678587, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.679345] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d84470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C756-0-0, dag_ret:0, dag_status:2, start_time:1727337665678587, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:05.679352] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=765, dag_cnt=4, dag_cnts_[dag.get_type()]=4, &dag=0x7f9264d84470, dag={ObIDag:{this:0x7f9264d84470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C756-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665678587, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.679359] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C756-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665678584, task_id:YB42C0A8002D-0006230056D8C756-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.679374] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.679376] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.679398] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=0] partition merge iter row count(i=0, row_count=141, ghost_row_count=0, pkey={tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337665635924, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.679417] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665679157, end_time=1727337665679416, runtime=259, *this={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C754-0-0, dag_ret:0, dag_status:2, start_time:1727337665678574, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.679425] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=851, dag_cnt=3, dag_cnts_[dag.get_type()]=3, &dag=0x7f9264d83a10, dag={ObIDag:{this:0x7f9264d83a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C754-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665678574, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.679432] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C754-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665678569, task_id:YB42C0A8002D-0006230056D8C754-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.679456] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337665635939, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.679464] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=8] update column cnt(column_cnt_=29, store_column_cnt_=29, column_cnt_=29, column_array_.count()=0) [2024-09-26 16:01:05.679467] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9287db3490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.679480] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.679484] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.679492] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:329}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.679527] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=29] succeed to build merge ctx(tablet_id={id:329}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9287db3490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:329}, table_count:1, [{i:0, table_key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e089f20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665678800, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=690us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.679553] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d83f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C755-0-0, dag_ret:0, dag_status:2, start_time:1727337665678580, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db35f0, output_block_cnt_arr:0x7f9287db35f8, concurrent_cnt:1, estimate_row_cnt:62, estimate_occupy_size:0, latest_update_ts:1727337665678800, estimated_finish_time:1727337682679553}) [2024-09-26 16:01:05.679560] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d84300, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C755-0-0, dag_ret:0, dag_status:2, start_time:1727337665678580, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.679565] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=5] task finish process(ret=0, start_time=1727337665678798, end_time=1727337665679565, runtime=767, *this={this:0x7f9264d84300, type:15, status:2, dag:{ObIDag:{this:0x7f9264d83f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C755-0-0, dag_ret:0, dag_status:2, start_time:1727337665678580, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.679584] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=2] schedule one task(task=0x7f9287cf8030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.679620] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=329, range=0x7f92aa752200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3a060, count:6, allocator:0x7f9269f787d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f9263a3a390, local_datums:0x7f9263a3a390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:01:05.679724] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.679773] ERROR issue_dba_error (ob_log.cpp:1875) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.679777] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.679786] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=0] partition merge iter row count(i=0, row_count=47, ghost_row_count=0, pkey={tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337665635939, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.679804] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665679595, end_time=1727337665679804, runtime=209, *this={this:0x7f9287cf8030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d83f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C755-0-0, dag_ret:0, dag_status:2, start_time:1727337665678580, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.679812] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1232, dag_cnt=2, dag_cnts_[dag.get_type()]=2, &dag=0x7f9264d83f40, dag={ObIDag:{this:0x7f9264d83f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C755-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665678580, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.679819] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C755-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665678577, task_id:YB42C0A8002D-0006230056D8C755-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.680299] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337665635895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.680314] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=14] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=15, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.680332] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337665635970, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:05.680341] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.680346] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.680342] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9287d2f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:05.680362] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:05.680352] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:452}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:17, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.680365] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:05.680371] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287d2f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101067}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:05.680381] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:452}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:452}, table_count:1, [{i:0, table_key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8652d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665678161, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.19ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.680391] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:101067}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9287d2f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101067}, table_count:1, [{i:0, table_key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b86a2d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337665678636, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.73ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:05.680408] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C752-0-0, dag_ret:0, dag_status:2, start_time:1727337665678011, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3480, output_block_cnt_arr:0x7f92728f3488, concurrent_cnt:1, estimate_row_cnt:10, estimate_occupy_size:0, latest_update_ts:1727337665678161, estimated_finish_time:1727337682680408}) [2024-09-26 16:01:05.680415] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264d83370, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C752-0-0, dag_ret:0, dag_status:2, start_time:1727337665678011, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.680420] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=5] task finish process(ret=0, start_time=1727337665678160, end_time=1727337665680420, runtime=2260, *this={this:0x7f9264d83370, type:15, status:2, dag:{ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C752-0-0, dag_ret:0, dag_status:2, start_time:1727337665678011, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.680420] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264d849a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C757-0-0, dag_ret:0, dag_status:2, start_time:1727337665678592, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287d2f4a0, output_block_cnt_arr:0x7f9287d2f4a8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337665678636, estimated_finish_time:1727337682680420}) [2024-09-26 16:01:05.680426] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264d84d60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d849a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C757-0-0, dag_ret:0, dag_status:2, start_time:1727337665678592, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.680441] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=15] task finish process(ret=0, start_time=1727337665678635, end_time=1727337665680441, runtime=1806, *this={this:0x7f9264d84d60, type:15, status:2, dag:{ObIDag:{this:0x7f9264d849a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C757-0-0, dag_ret:0, dag_status:2, start_time:1727337665678592, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.680455] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=5] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.680459] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=3] schedule one task(task=0x7f9264d84ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:05.680495] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=452, range=0x7f92ab056200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:15, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:17}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c6e060, count:5, allocator:0x7f926368c7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:17, datum_buffer:{capacity:32, datums:0x7f9262c6e360, local_datums:0x7f9262c6e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,]}) [2024-09-26 16:01:05.680526] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101067, range=0x7f92a9a56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9262c76060, count:8, allocator:0x7f926368e7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9262c763e0, local_datums:0x7f9262c763e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:05.680556] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=35][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:05.680588] ERROR issue_dba_error (ob_log.cpp:1875) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.680591] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.680599] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=0] partition merge iter row count(i=0, row_count=10, ghost_row_count=0, pkey={tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337665635895, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.680618] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665680467, end_time=1727337665680618, runtime=151, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C752-0-0, dag_ret:0, dag_status:2, start_time:1727337665678011, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.680627] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2616, dag_cnt=1, dag_cnts_[dag.get_type()]=1, &dag=0x7f9264d82fb0, dag={ObIDag:{this:0x7f9264d82fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C752-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665678011, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.680634] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C752-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665678004, task_id:YB42C0A8002D-0006230056D8C752-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.680758] ERROR issue_dba_error (ob_log.cpp:1875) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:05.680762] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:05.680770] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=0] partition merge iter row count(i=0, row_count=227, ghost_row_count=0, pkey={tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337665635970, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:05.680791] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337665680468, end_time=1727337665680790, runtime=322, *this={this:0x7f9264d84ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264d849a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C757-0-0, dag_ret:0, dag_status:2, start_time:1727337665678592, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:05.680799] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2207, dag_cnt=0, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264d849a0, dag={ObIDag:{this:0x7f9264d849a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C757-0-0, dag_ret:-4184, dag_status:5, start_time:1727337665678592, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:05.680806] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C757-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337665678590, task_id:YB42C0A8002D-0006230056D8C757-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:05.695784] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:05.695794] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.720846] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:05.720861] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337665720842) [2024-09-26 16:01:05.720878] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337665520353, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:05.720899] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223394956379, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.720905] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:05.720923] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:05.720932] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:05.720935] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:05.720942] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.720944] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.720946] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337665720882) [2024-09-26 16:01:05.725883] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:05.725894] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.756967] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:05.756978] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.769746] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337665355017]}) [2024-09-26 16:01:05.775393] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:05.775516] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=52, refresh_score_time=15, wash_time=2) [2024-09-26 16:01:05.776127] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=283113, clean_num=31457) [2024-09-26 16:01:05.787255] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:01:05.787292] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.800930] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:01:05.802918] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:05.817547] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:05.817558] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.821469] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.821474] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:05.821483] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.821489] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.821474] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.821484] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337665821470) [2024-09-26 16:01:05.821493] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.821506] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337665720881, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:05.821508] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.821508] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.821516] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.821516] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.821523] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.821527] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223395056949, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.821535] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:05.821545] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:05.821552] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:05.821563] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:05.821571] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.821575] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.821580] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337665821510) [2024-09-26 16:01:05.822445] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:05.847739] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:05.847755] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.856716] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:05.856728] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337665856714], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:05.856735] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337665856714], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:05.870299] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337665856714]}) [2024-09-26 16:01:05.878101] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:05.878110] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.878151] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=24][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666806530, now=[mts=1727337664876758]) [2024-09-26 16:01:05.878157] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666806530, now=[mts=1727337664876758], now0=[mts=1727337664876758], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:05.878163] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=4][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a312d100, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500860, assoc_session_id:3221500860, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337664876116, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:05.878179] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=15][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a2319bf8, id:3221500860, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312d100}) [2024-09-26 16:01:05.878187] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500860, *tx_desc={this:0x7f92a312d100, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500860, assoc_session_id:3221500860, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337664876116, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:05.878194] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500860, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a312d100, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500860, assoc_session_id:3221500860, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337664876116, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337664876735, use_das=false, nested_level=0, session={this:0x7f92a2319bf8, id:3221500860, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312d100}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:05.878219] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=24][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:05.878225] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=6][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:05.878228] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:05.878231] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:05.878235] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:05.878238] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:05.878244] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5] already timeout, do not need sleep(sleep_us=0, remain_us=998286, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337666876530) [2024-09-26 16:01:05.878247] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:05.878255] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:05.878257] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:05.878259] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:05.878264] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:05.878287] INFO [SQL.PC] need_check_schema_version (ob_plan_cache_value.cpp:2169) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2] need check schema(new_schema_version=1, cached_tenant_schema_version=1, is_contain_synonym()=false, contain_sys_name_table_=true, is_contain_tmp_tbl()=false, is_contain_sys_pl_object()=false, pc_ctx.sql_ctx_.session_info_->get_has_temp_table_flag()=false, need_check=true, constructed_sql_=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = ? ORDER BY row_id, column_name) [2024-09-26 16:01:05.878308] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:01:05.878311] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:05.891789] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=7] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-09-26 16:01:05.891801] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=8] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=693) [2024-09-26 16:01:05.908373] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:01:05.908381] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.922037] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:05.922055] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337665922034) [2024-09-26 16:01:05.922060] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337665821510, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:05.922088] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=223395157595, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:05.922103] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:05.922111] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:05.922117] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:05.922119] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:05.922125] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.922127] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:05.922129] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337665922075) [2024-09-26 16:01:05.938638] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:05.938648] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.942572] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1727337665942571, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006987, add_bytes: 0 [2024-09-26 16:01:05.945556] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [2925][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=12] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f92b77d02b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-09-26 16:01:05.964063] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:100, tid:2892}]) [2024-09-26 16:01:05.968975] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:05.968985] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:05.970962] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337665856714]}) [2024-09-26 16:01:05.976875] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:05.976978] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=49, refresh_score_time=15, wash_time=1) [2024-09-26 16:01:05.977609] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=314570, clean_num=31457) [2024-09-26 16:01:05.990602] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:05.999211] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:05.999220] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.001161] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:01:06.022088] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:06.022099] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337666022085) [2024-09-26 16:01:06.022105] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337665922075, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:06.022132] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223395258437, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:06.022138] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:06.022146] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:06.022152] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:06.022155] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:06.022162] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.022164] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.022169] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337666022108) [2024-09-26 16:01:06.028938] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593C-0-0] [lt=63][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:06.028946] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593C-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:06.028955] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593C-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:06.028960] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593C-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:06.029347] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:06.029355] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.042343] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500865, support_ssl=false) [2024-09-26 16:01:06.042378] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221500865, sess.client_addr_="127.0.0.1:36392") [2024-09-26 16:01:06.042384] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5] accept one succ(*s={this:0x7f9264e04030, session_id:3221500865, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:06.042450] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:06.042475] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=5] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:06.042511] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA6-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:06.042682] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA6-0-0] [lt=6][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:06.042692] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA6-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"weu;elplN!Yz|!)K|Q`o"}, ret=-4043) [2024-09-26 16:01:06.042698] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA6-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:06.042701] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA6-0-0] [lt=2][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:06.042708] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA6-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:06.042730] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA6-0-0] [lt=4] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500865, proxy_sessid:0}) [2024-09-26 16:01:06.042734] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA6-0-0] [lt=4] mark session id unused(sessid=3221500865) [2024-09-26 16:01:06.042738] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA6-0-0] [lt=2][errcode=0] server close connection(sessid=3221500865, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:06.042741] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA6-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500865, proxy_sessid=0) [2024-09-26 16:01:06.042744] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA6-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:06.042747] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA6-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500865, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:06.042779] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f9264e04030, session_id:3221500865, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337666041468, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:06.042792] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] kill and revert session(conn.sessid_=3221500865, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:06.042797] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] can close safely, do destroy(*s={this:0x7f9264e04030, session_id:3221500865, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337666041468, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:06.042802] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] connection close(sessid=3221500865, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:06.059777] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:06.059785] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.067928] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:06.067939] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:01:06.067943] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:01:06.067947] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:01:06.067950] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:06.067952] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:06.067955] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:01:06.067958] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:06.067960] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:06.067962] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:06.067964] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:06.067966] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:06.067968] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:06.067970] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:06.067976] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:06.067979] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:06.067981] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:06.067984] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:06.067990] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:01:06.067993] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:06.067996] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:06.068002] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:06.068008] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:06.068010] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:06.068011] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:06.068017] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:06.068020] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:06.068024] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589B-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:01:06.068027] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:06.068030] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:06.068034] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589B-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:06.068037] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589B-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337666067858, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:06.068041] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589B-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:06.068043] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589B-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:01:06.068065] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589B-0-0] [lt=3][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:01:06.068068] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589B-0-0] [lt=2][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:01:06.071471] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337665856714]}) [2024-09-26 16:01:06.074434] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=10] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:06.074691] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:06.074699] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:06.074703] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:06.074706] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:06.074708] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=1][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:06.074711] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:06.074715] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:06.074717] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:06.074719] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:06.074721] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:06.074722] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:06.074724] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:06.074726] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:06.074728] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:06.074735] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=5][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:06.074737] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:06.074740] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:06.074742] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:06.074744] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:01:06.074747] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:06.074751] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:06.074755] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:06.074759] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:06.074763] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:06.074765] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:06.074770] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:06.074773] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:06.074776] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2579][RSAsyncTask0][T0][YB42C0A8002D-0006230057584E74-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:01:06.074780] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:06.074783] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:06.074785] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:06.074787] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337666074649, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:06.074789] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:06.074833] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:01:06.090039] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:06.090049] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.092014] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=11] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:06.120155] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:01:06.120164] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.122301] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:06.122308] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:06.122315] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337666122298) [2024-09-26 16:01:06.122319] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337666022108, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:06.122333] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:06.122348] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:06.122351] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:06.122638] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:06.122648] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=8][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:06.122652] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:06.122669] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:06.122673] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:06.122675] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:06.122679] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:06.122681] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:06.122685] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:06.122687] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:06.122689] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:06.122691] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:06.122693] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:06.122695] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:06.122700] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:06.122703] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:06.122708] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=4][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:06.122710] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:06.122713] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:06.122726] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:06.122729] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:06.122734] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:06.122739] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:06.122741] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:06.122743] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:06.122748] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:06.122751] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:06.122754] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885782-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:06.122756] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:06.122759] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:06.122761] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:06.122764] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337666122584, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:06.122767] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:06.122770] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:06.122792] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:06.122795] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337666122795, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=451, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:06.122802] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:06.122805] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:06.122821] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223395357715, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:06.122826] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:06.122832] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:06.122837] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:06.122840] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:06.122845] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.122847] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.122849] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337666122810) [2024-09-26 16:01:06.150587] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:06.150596] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.172085] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=23] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337665856714]}) [2024-09-26 16:01:06.177992] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: SHARED_BLOCK_ReaderWriter_IO, mode: read, size: 4096.00, iops: 52.68, rt: 2779.00) [2024-09-26 16:01:06.178004] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:06.178036] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 10804.00) [2024-09-26 16:01:06.178041] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:06.178047] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:06.178051] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:06.178697] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:06.178800] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=56, refresh_score_time=15, wash_time=1) [2024-09-26 16:01:06.179404] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=346027, clean_num=31457) [2024-09-26 16:01:06.180885] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:06.180894] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.180902] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:06.201386] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:01:06.202371] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:01:06.202379] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=6] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:01:06.202386] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:01:06.211129] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:06.211139] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.222588] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:06.222600] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337666222586) [2024-09-26 16:01:06.222605] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337666122322, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:06.222620] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223395457447, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:06.222626] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:06.222632] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:06.222637] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:06.222640] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:06.222644] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.222646] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.222647] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337666222608) [2024-09-26 16:01:06.241420] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:06.241429] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.262280] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:06.262294] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:06.262299] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:06.262313] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:06.262318] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:06.262322] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:06.262325] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:06.262327] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:06.262330] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:06.262335] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=5][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:06.262338] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=3][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:06.262340] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:06.262342] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:06.262345] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:06.262351] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:06.262354] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:06.262357] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:06.262360] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:06.262363] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:01:06.262365] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:06.262368] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:06.262373] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:06.262379] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:06.262381] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:06.262383] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:06.262388] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:06.262392] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:06.262394] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:01:06.262399] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:06.262402] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:06.262404] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:06.262406] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337666262214, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:06.262410] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:06.262413] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:06.262443] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=4][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:06.262449] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=5][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:01:06.262451] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:06.262454] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:06.262471] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=1][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:06.262475] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=4][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:06.262477] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AA-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:06.271719] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:01:06.271733] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.272682] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337665856714]}) [2024-09-26 16:01:06.272697] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D9D2-0-0] [lt=11][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337666272676}) [2024-09-26 16:01:06.301993] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:06.302002] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.322480] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666251596, now=[mts=1727337665321546]) [2024-09-26 16:01:06.322492] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=12][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666251596, now=[mts=1727337665321546], now0=[mts=1727337665321546], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:06.322501] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269166d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337665321355, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:06.322536] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=34][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269166d80}) [2024-09-26 16:01:06.322547] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=11][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269166d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337665321355, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:06.322559] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=11] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269166d80, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337665321355, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337664321823, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269166d80}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:06.322581] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=21][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:06.322587] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:06.322606] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=18][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:06.322610] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:06.322613] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:06.322619] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:06.322624] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:01:06.322632] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:06.322635] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:06.322637] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:01:06.322643] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:01:06.322648] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=4] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000825) [2024-09-26 16:01:06.322652] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:01:06.322656] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:06.322660] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:01:06.322664] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:01:06.322668] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:01:06.322675] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=5][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:06.322717] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:01:06.322722] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:01:06.322719] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:06.322727] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:01:06.322728] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337666322716) [2024-09-26 16:01:06.322730] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:06.322733] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337666222608, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:06.322735] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:06.322738] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=3][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:01:06.322752] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] get wrs ts(ls_id={id:1}, delta=223395558151, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:06.322758] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:06.322764] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:06.322770] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:06.322773] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:06.322778] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.322780] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.322782] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337666322739) [2024-09-26 16:01:06.322855] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C83-0-0] [lt=4][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:01:06.322880] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:01:06.322885] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:01:06.322906] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=3] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:06.322911] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:01:06.323270] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:01:06.323280] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=10][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:06.332265] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:06.332272] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.357756] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.357768] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337666357754], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:06.357775] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337666357754], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:06.357779] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=5075, get_gts_with_stc_cnt=11875, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-09-26 16:01:06.357783] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:06.357786] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:06.362545] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:06.362551] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.373288] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337666357754]}) [2024-09-26 16:01:06.380230] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:06.380358] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash time detail, (compute_wash_size_time=55, refresh_score_time=15, wash_time=2) [2024-09-26 16:01:06.380982] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=377484, clean_num=31457) [2024-09-26 16:01:06.391837] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:06.392829] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:06.392836] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.394778] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=8] current monitor number(seq_=-1) [2024-09-26 16:01:06.401624] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:01:06.404541] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337665404477]) [2024-09-26 16:01:06.404552] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337665404477], now0=[mts=1727337665404477], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:06.404558] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337665403395, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:06.404576] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:01:06.404594] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=18][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337665403395, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:06.404603] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337665403395, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:06.404621] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:06.404626] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:06.404629] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:06.404632] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:06.404634] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:01:06.404638] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=28, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:06.404643] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] will sleep(sleep_us=28000, remain_us=589689, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=28, v.err_=-4283, timeout_timestamp=1727337666994332) [2024-09-26 16:01:06.423095] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:06.423104] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.423110] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:06.423126] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337666423099) [2024-09-26 16:01:06.423132] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337666322739, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:06.423150] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223395658621, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:06.423159] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:06.423166] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:06.423172] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:06.423177] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:06.423182] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.423184] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.423186] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337666423135) [2024-09-26 16:01:06.432851] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:28, local_retry_times:28, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:06.432865] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:06.432868] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:06.432870] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:06.432898] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:06.432937] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=72) [2024-09-26 16:01:06.432942] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:06.440999] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D444D-0-0] [lt=15][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:06.453192] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:06.453201] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.468019] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.468029] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.468047] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=17][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.468043] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.468044] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.468051] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.468051] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.468057] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.468056] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.469018] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.469023] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.469026] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.469028] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.469028] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.469033] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.469982] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.469997] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.469997] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.470001] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.470004] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.470000] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.470010] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=27][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.470012] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.470014] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.470963] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.470963] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.470994] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.470997] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.470972] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.470982] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.470999] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.471000] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=18][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.471000] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=28][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.471963] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.471969] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.471972] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.471979] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.471982] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.471984] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.471985] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.471989] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.471993] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.472906] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.472907] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.472918] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.472921] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.472928] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.472931] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:06.472944] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.473895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.473916] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.473905] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337666357754]}) [2024-09-26 16:01:06.473921] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.474862] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.474871] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.474888] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.475831] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.475838] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.475848] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.476806] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.476811] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.476818] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.477824] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.477840] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.477843] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.479799] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.479802] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.479802] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.479818] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=7] PNIO [ratelimit] time: 1727337666479817, bytes: 14727706, bw: 0.002960 MB/s, add_ts: 1000273, add_bytes: 3105 [2024-09-26 16:01:06.480720] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.480720] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.480720] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.481714] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.481716] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.481750] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.482703] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.482705] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.482722] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.483643] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.483652] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:06.483654] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.483659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.483657] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.484616] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.484636] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.484637] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.485591] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.485598] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.485634] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.486615] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.486640] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.486643] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.487566] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.487593] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.487595] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.488546] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.488558] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.488562] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.489076] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.489076] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.489077] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.490495] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.490498] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.490508] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.491018] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.491066] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.491069] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.492459] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.492463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.492470] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.493471] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.493482] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.493482] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.494418] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.494418] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.494422] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.495371] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.495408] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.495410] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.496338] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.496351] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.496350] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=35][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.497308] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.497336] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.497338] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.498296] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.499270] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.499277] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=25][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.499298] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.500272] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.500273] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.500275] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.501238] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.501240] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.501250] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.502190] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:06.514973] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:06.514981] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.523717] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:06.523729] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337666523714) [2024-09-26 16:01:06.523734] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337666423135, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:06.523752] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223395759214, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:06.523758] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:06.523766] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:06.523771] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:06.523774] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:06.523779] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.523782] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.523784] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337666523737) [2024-09-26 16:01:06.525767] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593D-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:06.525777] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593D-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:06.525785] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593D-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:06.525790] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593D-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:06.545221] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:06.545230] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.550101] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=1, size_used=6672, mem_used=12478464) [2024-09-26 16:01:06.566671] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=9] gc stale ls task succ [2024-09-26 16:01:06.574487] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337666357754]}) [2024-09-26 16:01:06.575465] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:06.575501] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.581434] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:06.581550] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=51, refresh_score_time=28, wash_time=1) [2024-09-26 16:01:06.582156] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=408941, clean_num=31457) [2024-09-26 16:01:06.601838] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=4] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.601850] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.601856] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.601935] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:01:06.601953] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=5] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.605780] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:06.605790] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.621393] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:06.624340] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:06.624360] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337666624337) [2024-09-26 16:01:06.624366] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337666523736, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:06.624371] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:06.624379] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:06.624381] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:06.624697] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:06.624705] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:06.624709] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:06.624712] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:06.624715] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:06.624717] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:06.624721] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:06.624723] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:06.624744] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=21][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:06.624746] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:06.624748] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:06.624750] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:06.624753] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:06.624755] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:06.624761] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:06.624763] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:06.624766] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:06.624769] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:06.624771] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:06.624774] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:06.624776] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:06.624782] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:06.624787] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:06.624789] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:06.624790] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:06.624796] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:06.624799] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:06.624801] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885783-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:06.624804] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:06.624807] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:06.624809] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:06.624811] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337666624618, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:06.624814] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:06.624817] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:06.624845] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:06.624848] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337666624847, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=472, wlock_time=8, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:06.624854] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:06.624856] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:06.624871] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223395860791, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:06.624875] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:06.624881] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:06.624885] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:06.624887] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:06.624891] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.624893] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.624895] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337666624860) [2024-09-26 16:01:06.636024] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:06.636033] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.663476] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855F9-0-0] [lt=3] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:06.665350] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=4] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:06.666291] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:06.666298] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.668282] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4184, dropped:492, tid:2864}]) [2024-09-26 16:01:06.675097] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337666357754]}) [2024-09-26 16:01:06.696602] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:06.696622] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.724587] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:06.724610] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337666724582) [2024-09-26 16:01:06.724616] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337666624368, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:06.724633] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223395960399, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:06.724638] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:06.724646] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:06.724651] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:06.724654] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:06.724659] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.724662] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.724664] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337666724619) [2024-09-26 16:01:06.726657] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:06.726665] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.739578] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=15] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:01:06.757102] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:06.757111] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.775674] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337666357754]}) [2024-09-26 16:01:06.782654] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:06.782767] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=61, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:06.783392] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=440398, clean_num=31457) [2024-09-26 16:01:06.787460] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:01:06.787468] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.802396] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=943680, replace_num=15728) [2024-09-26 16:01:06.802407] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:01:06.810878] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:06.817567] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:06.817576] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.821555] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.821570] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.821579] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.821578] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=9] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.821590] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.821595] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.821600] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.821606] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.821614] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.822524] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.822536] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:06.824748] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:06.824762] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337666824744) [2024-09-26 16:01:06.824780] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337666724619, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:06.824803] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223396061036, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:06.824813] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:06.824821] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:06.824826] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:06.824829] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:06.824836] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.824838] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.824839] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337666824785) [2024-09-26 16:01:06.827502] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [2915][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=9] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-09-26 16:01:06.847644] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:06.847657] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.858832] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:06.858843] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337666858828], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:06.858851] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337666858828], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:06.877375] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337666858828]}) [2024-09-26 16:01:06.878266] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:06.878275] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.880199] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=4][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666806530, now=[mts=1727337665878307]) [2024-09-26 16:01:06.880219] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=20][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666806530, now=[mts=1727337665878307], now0=[mts=1727337665878307], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:06.880226] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a312d100, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500860, assoc_session_id:3221500860, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337665878187, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:06.880244] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a2319bf8, id:3221500860, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312d100}) [2024-09-26 16:01:06.880254] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=10][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500860, *tx_desc={this:0x7f92a312d100, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500860, assoc_session_id:3221500860, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337665878187, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:06.880262] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500860, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a312d100, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500860, assoc_session_id:3221500860, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337665878187, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337664876735, use_das=false, nested_level=0, session={this:0x7f92a2319bf8, id:3221500860, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312d100}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:06.880291] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=28][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:06.880297] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:06.880300] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:06.880303] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:06.880306] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:06.880310] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:06.880315] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:01:06.880322] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:06.880324] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:06.880326] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:01:06.880331] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:01:06.880335] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2003600) [2024-09-26 16:01:06.880338] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508D-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:01:06.880341] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:06.880345] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:01:06.880347] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:01:06.880358] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8620, start=1727337664876732, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:06.880362] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:01:06.880365] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:06.880402] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:01:06.880405] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:01:06.880407] WDIAG [SHARE] get_snapshot_gc_scn (ob_global_stat_proxy.cpp:164) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:01:06.880410] WDIAG [STORAGE] get_global_info (ob_tenant_freeze_info_mgr.cpp:811) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to get global info(ret=-4012, tenant_id=1) [2024-09-26 16:01:06.880413] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:954) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] failed to get global info(ret=-4012) [2024-09-26 16:01:06.880418] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1008) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] fail to try update info(tmp_ret=-4012, tmp_ret="OB_TIMEOUT") [2024-09-26 16:01:06.908518] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:06.908527] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.926123] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:06.926151] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337666926120) [2024-09-26 16:01:06.926157] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337666824784, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:06.926184] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223396162628, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:06.926192] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:06.926200] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:06.926205] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:06.926208] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:06.926214] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.926217] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:06.926219] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337666926161) [2024-09-26 16:01:06.937172] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C44-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2024-09-26 16:01:06.937186] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C44-0-0] [lt=13][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2024-09-26 16:01:06.937191] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C44-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2024-09-26 16:01:06.937194] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C44-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2024-09-26 16:01:06.937198] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C44-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:06.937200] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C44-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:06.937204] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C44-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2024-09-26 16:01:06.937206] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C44-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:06.937238] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2904][T1_DupTbLease][T1][YB42C0A8002D-0006230057084C44-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:06.938809] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:06.938817] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.942732] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337666942731, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000160, add_bytes: 0 [2024-09-26 16:01:06.969066] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:01:06.969076] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:06.977876] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337666858828]}) [2024-09-26 16:01:06.983869] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:06.983972] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=51, refresh_score_time=15, wash_time=1) [2024-09-26 16:01:06.984569] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=471855, clean_num=31457) [2024-09-26 16:01:06.991603] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] tx gc loop thread is running(MTL_ID()=1) [2024-09-26 16:01:06.991610] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] try gc retain ctx [2024-09-26 16:01:06.992003] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=2] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-09-26 16:01:06.992032] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:06.999387] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=19] success to notify_need_writing_throttling True [2024-09-26 16:01:06.999395] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.003263] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=4] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:01:07.026771] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:07.026786] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337667026767) [2024-09-26 16:01:07.026792] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337666926161, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:07.026813] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223396263270, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:07.026830] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:07.026838] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:07.026844] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:07.026847] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:07.026854] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.026856] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.026858] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337667026795) [2024-09-26 16:01:07.026939] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593E-0-0] [lt=34][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:07.026949] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593E-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:07.026958] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593E-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:07.026963] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593E-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:07.029679] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:07.029686] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.044496] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500870, support_ssl=false) [2024-09-26 16:01:07.044519] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221500870, sess.client_addr_="127.0.0.1:36394") [2024-09-26 16:01:07.044525] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f9272a04030, session_id:3221500870, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:07.044560] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:07.044580] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:07.044617] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA8-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:07.044785] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA8-0-0] [lt=3][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:07.044794] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA8-0-0] [lt=8][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:",*.H^!&=@jt+r"K7pP!O"}, ret=-4043) [2024-09-26 16:01:07.044800] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA8-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:07.044815] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA8-0-0] [lt=14][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:07.044823] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA8-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:07.044845] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA8-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500870, proxy_sessid:0}) [2024-09-26 16:01:07.044849] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA8-0-0] [lt=4] mark session id unused(sessid=3221500870) [2024-09-26 16:01:07.044853] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA8-0-0] [lt=3][errcode=0] server close connection(sessid=3221500870, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:07.044857] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA8-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500870, proxy_sessid=0) [2024-09-26 16:01:07.044859] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA8-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:07.044862] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA8-0-0] [lt=3] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500870, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:07.044903] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4015] close sql sock by user req(*s={this:0x7f9272a04030, session_id:3221500870, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337667044298, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:07.044914] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221500870, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:07.044917] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9272a04030, session_id:3221500870, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337667044298, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:07.044921] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500870, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:07.059913] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:07.059921] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.069017] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:07.069029] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:01:07.069033] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:01:07.069051] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:01:07.069055] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:07.069057] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:07.069061] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:01:07.069063] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:07.069065] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:07.069067] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:07.069069] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:07.069071] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:07.069074] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:07.069076] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:07.069083] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:07.069086] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.069090] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=3][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:07.069103] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589C-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:07.070698] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=6][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:50, tid:2464}]) [2024-09-26 16:01:07.075537] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=8] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:07.075818] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:07.075828] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:07.075832] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:07.075835] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:07.075846] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:07.075849] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:07.075852] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:07.075854] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:07.075856] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:07.075857] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:07.075859] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:07.075861] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:07.075863] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:07.075865] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:07.075870] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:07.075871] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.075874] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.075876] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:07.075878] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:01:07.075880] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:07.075882] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:07.075888] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:07.075894] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:07.075896] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:07.075897] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:07.075902] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:07.075906] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E76-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:07.075909] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2579][RSAsyncTask0][T0][YB42C0A8002D-0006230057584E76-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:01:07.075912] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:07.075915] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:07.075920] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:07.075922] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337667075774, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:07.075925] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:07.075986] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:01:07.078470] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337666858828]}) [2024-09-26 16:01:07.090228] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:07.090235] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.092159] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=9] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:07.120461] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:07.120472] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.127317] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:07.127326] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:07.127343] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337667127314) [2024-09-26 16:01:07.127365] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337667026795, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:07.127371] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:07.127380] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:07.127383] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:07.127682] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:07.127692] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:07.127696] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:07.127699] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:07.127702] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:07.127705] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:07.127708] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:07.127711] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:07.127712] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:07.127714] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:07.127716] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:07.127718] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:07.127720] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:07.127722] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:07.127727] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:07.127729] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.127731] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.127734] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:07.127739] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:07.127742] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:07.127744] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:07.127749] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:07.127754] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:07.127757] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:07.127758] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:07.127764] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:07.127768] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:07.127771] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885784-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:07.127774] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:07.127776] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:07.127778] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:07.127781] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337667127618, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:07.127784] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:07.127787] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:07.127813] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:07.127818] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337667127818, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=442, wlock_time=11, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:07.127825] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:07.127828] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:07.127844] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223396363793, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:07.127849] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:07.127856] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:07.127860] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:07.127863] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:07.127870] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.127872] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.127874] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337667127832) [2024-09-26 16:01:07.150749] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:07.150767] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.179082] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337666858828]}) [2024-09-26 16:01:07.181054] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:07.181071] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=15][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.181080] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:07.183966] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 0.99, rt: 12144.00) [2024-09-26 16:01:07.183977] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:07.184003] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:07.184008] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:07.184726] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:07.184827] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:07.185433] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=503312, clean_num=31457) [2024-09-26 16:01:07.203519] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:01:07.211341] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:07.211351] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.213486] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:01:07.213499] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:07.213505] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:01:07.227927] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:07.227938] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337667227924) [2024-09-26 16:01:07.227943] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337667127368, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:07.227958] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223396463461, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:07.227964] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:07.227971] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:07.227976] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:07.227979] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:07.227984] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.227987] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.227989] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337667227946) [2024-09-26 16:01:07.241588] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=28] success to notify_need_writing_throttling True [2024-09-26 16:01:07.241595] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.263406] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:07.263417] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:07.263421] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:07.263424] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:07.263427] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:07.263429] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:07.263433] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:07.263435] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:07.263437] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:07.263438] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:07.263440] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:07.263442] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:07.263445] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:07.263447] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:07.263452] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:07.263454] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.263456] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.263458] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:07.263461] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:01:07.263463] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:07.263465] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:07.263470] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:07.263475] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:07.263477] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:07.263478] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:07.263483] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:07.263486] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:07.263489] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:01:07.263491] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:07.263494] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:07.263496] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:07.263498] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337667263331, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:07.263501] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:07.263503] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:07.263528] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:07.263530] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:01:07.263533] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=3][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:07.263535] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:07.263537] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=1][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:07.263540] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:07.263542] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AB-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:07.271847] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:07.271856] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.279512] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337666858828]}) [2024-09-26 16:01:07.279559] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D9DE-0-0] [lt=8][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337667279504}) [2024-09-26 16:01:07.302124] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:07.302133] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.324464] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337668252917, now=[mts=1727337666323269]) [2024-09-26 16:01:07.324475] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337668252917, now=[mts=1727337666323269], now0=[mts=1727337666323269], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:07.324482] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269168b20, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337666322789, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:07.324509] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=26][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926361e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269168b20}) [2024-09-26 16:01:07.324517] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269168b20, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337666322789, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:07.324524] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269168b20, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337666322789, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337666323237, use_das=false, nested_level=0, session={this:0x7f926361e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269168b20}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:07.324552] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=27][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:07.324557] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:07.324560] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:07.324566] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:07.324568] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:07.324572] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:07.324577] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=998340, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337668322917) [2024-09-26 16:01:07.324580] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:07.324587] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:07.324590] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:07.324591] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:07.324596] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:07.324641] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:01:07.324647] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:07.328495] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:07.328505] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337667328492) [2024-09-26 16:01:07.328511] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337667227946, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:07.328529] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223396564055, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:07.328534] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:07.328541] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:07.328546] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:07.328549] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:07.328554] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.328557] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.328559] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337667328514) [2024-09-26 16:01:07.332450] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:07.332455] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.359887] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.359895] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337667359885], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:07.359901] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337667359885], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:07.359904] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:07.359907] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:07.363690] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:07.363697] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.380286] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337667359885]}) [2024-09-26 16:01:07.385823] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:07.385926] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=48, refresh_score_time=16, wash_time=1) [2024-09-26 16:01:07.386531] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=534769, clean_num=31457) [2024-09-26 16:01:07.391987] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=8] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:07.394995] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:01:07.395001] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.403705] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:01:07.425058] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:07.425067] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.427181] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] ====== tenant freeze timer task ====== [2024-09-26 16:01:07.427190] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:01:07.427230] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337665635081, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337665635046, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337665635102, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:01:07.427254] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:01:07.427310] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E6-0-0] [lt=1] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:01:07.427320] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E6-0-0] [lt=7] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:01:07.427327] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E6-0-0] [lt=5] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:01:07.427331] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E6-0-0] [lt=3] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:01:07.427335] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E6-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:01:07.427338] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E6-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:01:07.427340] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E6-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:01:07.427350] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E6-0-0] [lt=10] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:01:07.427354] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E6-0-0] [lt=4][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:07.427357] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E6-0-0] [lt=3] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:07.427360] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E6-0-0] [lt=1][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:07.427405] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851E6-0-0] [lt=3] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:01:07.430118] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:07.430146] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337667430115) [2024-09-26 16:01:07.430152] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337667328514, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:07.430167] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223396666577, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:07.430173] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:07.430180] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:07.430184] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:07.430187] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:07.430192] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.430194] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.430196] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337667430154) [2024-09-26 16:01:07.430414] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=1] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:01:07.430426] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=7] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:07.430432] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=5][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:07.430435] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:07.430439] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:07.430444] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=4] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:01:07.430446] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:07.430448] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:07.430464] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:07.430467] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:07.430470] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:07.430472] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:07.430475] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:07.430477] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:07.430483] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:07.430485] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:07.430491] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:07.430494] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.430497] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.430500] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:07.430504] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-09-26 16:01:07.430510] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:07.430516] WDIAG [SERVER] do_process (obmp_query.cpp:771) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=5][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-09-26 16:01:07.430522] WDIAG [SERVER] do_process (obmp_query.cpp:903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, session={this:0x7f92a230bdf8, id:3221487638, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-09-26 16:01:07.430532] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F56-0-0] [lt=6] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:07.434008] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337666924332, now=[mts=1727337666432936]) [2024-09-26 16:01:07.434023] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337666924332, now=[mts=1727337666432936], now0=[mts=1727337666432936], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:07.434030] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337666432853, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:07.434045] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=14][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}) [2024-09-26 16:01:07.434051] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337666432853, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:07.434058] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c59ce378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92bc8ca660, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337666432853, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337636994336, use_das=false, nested_level=0, session={this:0x7f92bc8a6038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92bc8ca660}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:07.434074] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:07.434080] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:07.434082] WDIAG [SQL] open (ob_result_set.cpp:157) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:07.434085] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:07.434088] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:01:07.434091] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=29, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:07.434096] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:29, local_retry_times:29, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:01:07.434102] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:07.434104] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:07.434105] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:01:07.434111] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2580][RSAsyncTask1][T1][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:01:07.434115] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=3] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, process_time=30439778) [2024-09-26 16:01:07.434118] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:01:07.434122] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name) [2024-09-26 16:01:07.434125] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:01:07.434129] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=4][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:01:07.434132] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8900, start=1727337636994332, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name) [2024-09-26 16:01:07.434135] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:01:07.434138] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=1][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name) [2024-09-26 16:01:07.434165] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:01:07.434180] WDIAG [SHARE] load_refresh_schema_status (ob_schema_status_proxy.cpp:240) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=15][errcode=-4012] fail to load core table(ret=-4012) [2024-09-26 16:01:07.434183] INFO [SHARE] load_refresh_schema_status (ob_schema_status_proxy.cpp:284) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=2] [SCHEMA_STATUS] load refreshed schema status(ret=-4012) [2024-09-26 16:01:07.434190] ERROR issue_dba_error (ob_log.cpp:1875) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4012, file="ob_root_service.cpp", line_no=10369, info="rs_monitor_check : fail to start root service") [2024-09-26 16:01:07.434202] WARN update_fail_count (ob_root_service.cpp:10374) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=7][errcode=-4752] rootservice start process has failure(msg="rootservice start()/do_restart() has failure", ret=-4012, ret="OB_TIMEOUT", fail_cnt=52) [2024-09-26 16:01:07.434229] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2580][RSAsyncTask1][T0][YB42C0A8002D-0006230057D84F46-0-0] [lt=2][errcode=-4012] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-4012) [2024-09-26 16:01:07.434242] INFO [SHARE] store (ob_rs_mgr.cpp:128) [2582][RSAsyncTask3][T0][Y0-0006230057684E61-0-0] [lt=4] store rs list succeed(agent=0, addr_list=[{server:"192.168.0.45:2882", role:1, sql_port:2881}], force=true) [2024-09-26 16:01:07.447981] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D44AC-0-0] [lt=11][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:07.455474] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:07.455483] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.468067] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.468079] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.468086] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.468736] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.468744] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.468750] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.469440] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.469446] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.469449] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.469965] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.469968] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.469982] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=14][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.470517] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.470521] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.470523] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.471059] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.471064] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.471067] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.471588] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.471592] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.471594] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.472153] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.472158] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.472161] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.472809] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.472813] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.472815] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.473381] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.473386] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.473389] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.473983] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.473987] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.473990] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.474651] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.474657] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.474660] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.475246] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.475249] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.475252] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.475836] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.475844] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.475849] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.476484] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.476488] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.476490] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.477175] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.477178] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.477180] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:07.477931] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.478704] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.479309] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.479918] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.480863] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.480872] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337667359885]}) [2024-09-26 16:01:07.481897] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.482856] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.483821] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.484784] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.485761] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:07.485772] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.485848] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.487744] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=6] PNIO [ratelimit] time: 1727337667487744, bytes: 14731963, bw: 0.004028 MB/s, add_ts: 1007927, add_bytes: 4257 [2024-09-26 16:01:07.487772] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.488767] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.489678] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.490692] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.492594] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.493172] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.494543] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.495057] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.495730] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.496524] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.497470] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.498502] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.499498] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.500403] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.501439] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.502374] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.503336] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.504314] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.505659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.506253] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.507297] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.508715] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.510223] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.512130] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.513132] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.514092] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.515061] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.516058] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.516062] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:07.516068] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.517994] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.519950] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.521083] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.521148] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593F-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:07.521155] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593F-0-0] [lt=7][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:07.521164] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593F-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:07.521169] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718593F-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:07.521941] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.522541] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.523909] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.524884] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.525837] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.526510] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.527770] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.528532] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.529698] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.530715] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:07.530727] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337667530711) [2024-09-26 16:01:07.530732] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337667430154, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:07.530725] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.530760] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223396767199, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:07.530779] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:07.530787] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:07.530792] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:07.530795] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:07.530803] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.530805] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.530807] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337667530745) [2024-09-26 16:01:07.531725] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=19][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.532658] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.533450] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.534597] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.535522] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.536545] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.537510] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.538467] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.539191] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.540248] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.540935] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.541468] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.542306] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.543376] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.544398] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.545414] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.546312] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.546326] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:07.546334] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.547313] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.548274] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.549244] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.549819] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.551259] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6682, mem_used=12478464) [2024-09-26 16:01:07.551282] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.552173] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.553184] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.554132] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.555103] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.555676] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.556490] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.557050] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.559012] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.560034] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.560940] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.561767] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:07.566824] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=16] gc stale ls task succ [2024-09-26 16:01:07.576710] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:07.576717] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.581487] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337667359885]}) [2024-09-26 16:01:07.587551] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:07.587652] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=1) [2024-09-26 16:01:07.588283] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=566226, clean_num=31457) [2024-09-26 16:01:07.601054] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.601066] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.601072] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.601669] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.603828] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:01:07.606939] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:07.606946] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.625476] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:07.630813] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:07.630825] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337667630811) [2024-09-26 16:01:07.630831] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337667530735, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:07.630836] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:07.630844] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:07.630847] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:07.631162] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:07.631172] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:07.631176] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:07.631179] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:07.631183] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:07.631185] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:07.631188] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:07.631190] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:07.631192] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:07.631194] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:07.631196] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:07.631198] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:07.631200] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:07.631202] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:07.631208] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:07.631210] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.631213] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.631224] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:07.631227] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:07.631230] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:07.631233] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:07.631239] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:07.631245] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:07.631248] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:07.631249] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:07.631255] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:07.631259] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:07.631262] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885785-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:07.631264] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:07.631268] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:07.631270] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:07.631273] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337667631086, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:07.631277] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:07.631280] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:07.631309] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:07.631314] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337667631314, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=473, wlock_time=8, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:07.631333] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:07.631336] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:07.631353] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223396866781, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:07.631357] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:07.631363] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:07.631368] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:07.631370] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:07.631376] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.631378] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.631381] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337667631340) [2024-09-26 16:01:07.637210] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:07.637218] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.646930] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=15] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:01:07.665452] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=7] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:07.667463] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:07.667470] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.682141] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=18] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337667359885]}) [2024-09-26 16:01:07.691154] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:45) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2] dump dtl interm result cost(us)(dump_cost=13961, ret=0, interm count=0, dump count=346) [2024-09-26 16:01:07.697683] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:07.697691] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.704658] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:60) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11] clear dtl interm result cost(us)(clear_cost=13492, ret=0, expire_keys_.count()=0, dump count=346, interm count=0, clean count=0) [2024-09-26 16:01:07.705048] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:07.705060] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:07.705064] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:07.705071] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=6][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:07.705075] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:07.705077] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:07.705080] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:07.705083] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:07.705085] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:07.705087] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:07.705090] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=3][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:07.705092] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:07.705094] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:07.705097] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=3][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:07.705105] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:07.705109] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.705124] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=14][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:07.705129] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:07.705132] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:01:07.705135] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:07.705137] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:07.705144] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:07.705149] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:07.705152] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:07.705153] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:07.705161] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD8-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:07.727987] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:07.727996] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.730914] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:07.730925] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337667730911) [2024-09-26 16:01:07.730931] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337667630834, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:07.730949] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223396967390, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:07.730955] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:07.730963] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:07.730969] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:07.730972] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:07.730979] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.730981] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.730983] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337667730935) [2024-09-26 16:01:07.758089] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:07.758099] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.782685] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337667359885]}) [2024-09-26 16:01:07.788531] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:07.788542] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.788661] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:07.788767] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Wash time detail, (compute_wash_size_time=58, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:07.789366] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=597683, clean_num=31457) [2024-09-26 16:01:07.804159] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:01:07.814983] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:07.818804] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:07.818811] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.821737] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.821746] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.821750] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.821755] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.821759] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.821763] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.821769] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.821869] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.821879] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.821886] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.822714] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:07.831295] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:07.831304] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337667831292) [2024-09-26 16:01:07.831310] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337667730934, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:07.831326] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223397066999, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:07.831333] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:07.831341] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:07.831346] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:07.831350] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:07.831356] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.831359] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.831361] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337667831313) [2024-09-26 16:01:07.849156] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:07.849162] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.861061] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:07.861072] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337667861059], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:07.861079] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337667861059], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:07.879371] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:07.879409] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=36][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.883270] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=24] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337667861059]}) [2024-09-26 16:01:07.909626] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:07.909636] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.931995] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:07.932008] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337667931991) [2024-09-26 16:01:07.932012] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337667831312, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:07.932026] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223397168376, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:07.932033] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:07.932040] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:07.932056] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:07.932060] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:07.932066] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.932069] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:07.932071] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337667932015) [2024-09-26 16:01:07.940868] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21] success to notify_need_writing_throttling True [2024-09-26 16:01:07.940873] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.949673] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337667949672, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006941, add_bytes: 0 [2024-09-26 16:01:07.971183] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:07.971190] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:07.983438] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=21] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337667861059]}) [2024-09-26 16:01:07.989848] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:07.989951] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:07.990558] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=629140, clean_num=31457) [2024-09-26 16:01:07.992224] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:08.001446] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:08.001454] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.004390] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:01:08.027200] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA8-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:08.027211] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA8-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:08.027239] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA8-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:08.027248] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:398) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=0] start to print blacklist info [2024-09-26 16:01:08.027272] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=2] ls blacklist refresh finish(cost_time=421) [2024-09-26 16:01:08.028946] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185940-0-0] [lt=33][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:08.028955] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185940-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:08.028965] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185940-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:08.028970] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185940-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:08.031510] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:08.031516] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.032339] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:08.032352] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337668032336) [2024-09-26 16:01:08.032358] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337667932015, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:08.032375] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223397268727, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:08.032380] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:08.032387] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:08.032391] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:08.032395] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:08.032401] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.032404] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.032406] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337668032361) [2024-09-26 16:01:08.047201] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] new mysql sessid created(conn.sessid_=3221500874, support_ssl=false) [2024-09-26 16:01:08.047232] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221500874, sess.client_addr_="127.0.0.1:36396") [2024-09-26 16:01:08.047238] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f9267204030, session_id:3221500874, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:08.047542] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:08.047554] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:08.049067] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA7-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:08.049310] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA7-0-0] [lt=8][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:08.049319] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA7-0-0] [lt=7][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"LojIiWV,t(gO5X@sFC\'"}, ret=-4043) [2024-09-26 16:01:08.049325] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA7-0-0] [lt=4][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:08.049329] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA7-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:08.049336] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA7-0-0] [lt=1] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:08.049359] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA7-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500874, proxy_sessid:0}) [2024-09-26 16:01:08.049363] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA7-0-0] [lt=4] mark session id unused(sessid=3221500874) [2024-09-26 16:01:08.049366] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA7-0-0] [lt=2][errcode=0] server close connection(sessid=3221500874, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:08.049369] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA7-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500874, proxy_sessid=0) [2024-09-26 16:01:08.049372] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA7-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:08.049374] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA7-0-0] [lt=1] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500874, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:08.049413] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f9267204030, session_id:3221500874, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337668047336, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:08.049437] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221500874, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:08.049441] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] can close safely, do destroy(*s={this:0x7f9267204030, session_id:3221500874, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337668047336, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:08.049446] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500874, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:08.061981] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:08.061992] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.070401] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589D-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:08.070442] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589D-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:08.076466] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:08.076825] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=16] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:08.076833] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:08.076837] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:08.076841] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:08.076844] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:08.076846] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:08.076850] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:08.076852] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:08.076854] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:08.076856] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:08.076858] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:08.076860] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:08.076862] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:08.076876] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E78-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:08.083601] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337667861059]}) [2024-09-26 16:01:08.092279] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=33][errcode=-4018] try free send task failed(ret=-4018) [2024-09-26 16:01:08.092368] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:08.092377] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.092398] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:08.122613] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:08.122620] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.132575] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:08.132584] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:08.132604] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337668132572) [2024-09-26 16:01:08.132610] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337668032361, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:08.132617] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:08.132627] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:08.132630] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:08.132959] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885786-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:08.132968] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885786-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:08.132997] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885786-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:08.133038] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337668133037, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=415, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:08.133049] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:08.133067] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223397369501, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:08.133072] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:08.133078] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:08.133082] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:08.133085] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:08.133090] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.133092] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.133094] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337668133054) [2024-09-26 16:01:08.153847] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:08.153859] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.175878] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:142, tid:2579}]) [2024-09-26 16:01:08.183920] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:08.183929] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.183941] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:08.183977] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337667861059]}) [2024-09-26 16:01:08.187520] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 9298.00) [2024-09-26 16:01:08.187529] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:08.187580] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:08.187585] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:08.190750] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:08.190856] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=54, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:08.191517] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=660597, clean_num=31457) [2024-09-26 16:01:08.203065] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=2] ====== check clog disk timer task ====== [2024-09-26 16:01:08.203075] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=7] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:01:08.203084] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:01:08.204573] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:01:08.214367] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:08.214379] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.232625] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:08.232638] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337668232621) [2024-09-26 16:01:08.232644] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337668132613, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:08.232662] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223397468343, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:08.232669] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:08.232677] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:08.232683] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:08.232686] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:08.232693] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.232696] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.232701] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337668232647) [2024-09-26 16:01:08.244518] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:08.244528] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.263959] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:08.263971] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:08.263975] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:08.263979] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:08.263983] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:08.263985] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:08.263988] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:08.263990] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:08.263992] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:08.263994] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:08.263996] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:08.263998] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:08.264001] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:08.264003] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:08.264009] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:08.264012] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:08.264014] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:08.264016] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:08.264019] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:01:08.264022] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:08.264024] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:08.264030] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:08.264035] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:08.264037] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:08.264039] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:08.264045] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:08.264047] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:08.264050] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:01:08.264052] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:08.264054] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:08.264056] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:08.264058] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337668263885, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:08.264062] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:08.264064] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:08.264114] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:08.264117] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=3][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:01:08.264128] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=11][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:08.264131] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=3][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:08.264134] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:08.264137] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:08.264139] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AC-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:08.274651] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:08.274662] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.284490] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337667861059]}) [2024-09-26 16:01:08.284499] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D9EA-0-0] [lt=8][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337668284484}) [2024-09-26 16:01:08.304779] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:08.304789] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.320039] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD9-0-0] [lt=1][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-09-26 16:01:08.320047] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD9-0-0] [lt=7][errcode=-4002] refresh sever list failed(ret=-4002) [2024-09-26 16:01:08.320049] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DD9-0-0] [lt=2][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-09-26 16:01:08.325186] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337668252917, now=[mts=1727337667324629]) [2024-09-26 16:01:08.325194] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337668252917, now=[mts=1727337667324629], now0=[mts=1727337667324629], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:08.325201] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269168b20, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337667323706, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:08.325218] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926361e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269168b20}) [2024-09-26 16:01:08.325226] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269168b20, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337667323706, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:08.325233] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269168b20, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337667323706, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337666323237, use_das=false, nested_level=0, session={this:0x7f926361e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269168b20}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:08.325250] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:08.325255] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:08.325257] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:08.325260] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:08.325262] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:08.325266] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:08.325270] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:01:08.325277] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:08.325278] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=1][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:08.325280] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:01:08.325285] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C84-0-0] [lt=1][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:01:08.325288] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2002050) [2024-09-26 16:01:08.325290] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:01:08.325293] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:08.325296] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:01:08.325298] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=1][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:01:08.325300] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:01:08.325305] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:08.325331] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:01:08.325333] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:01:08.325336] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:01:08.325338] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:08.325342] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:08.325344] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:01:08.325466] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C84-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:01:08.325473] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:01:08.325475] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:01:08.325484] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:08.325494] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=8][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:01:08.325782] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:01:08.325788] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:08.332764] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:08.332774] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337668332762) [2024-09-26 16:01:08.332779] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337668232646, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:08.332796] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223397568619, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:08.332802] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:08.332810] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:08.332815] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:08.332818] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:08.332824] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.332827] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.332828] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337668332782) [2024-09-26 16:01:08.334902] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:08.334911] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.361512] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=25] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:08.361525] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337668361509], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:08.361534] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337668361509], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:08.361537] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337668361509]}) [2024-09-26 16:01:08.361543] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:08.361547] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:08.364987] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:08.364999] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.385095] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337668361509]}) [2024-09-26 16:01:08.391665] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:08.391784] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=30, wash_time=2) [2024-09-26 16:01:08.392408] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=692054, clean_num=31457) [2024-09-26 16:01:08.392423] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=5] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:08.393034] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=11] current monitor number(seq_=-1) [2024-09-26 16:01:08.395118] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18] success to notify_need_writing_throttling True [2024-09-26 16:01:08.395126] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.404714] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:01:08.425236] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:08.425266] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.432850] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:08.432865] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337668432847) [2024-09-26 16:01:08.432870] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337668332782, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:08.432887] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223397668510, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:08.432892] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:08.432899] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:08.432904] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:08.432907] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:08.432912] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.432914] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.432916] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337668432873) [2024-09-26 16:01:08.453912] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:33) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=3] === Run print tenant memory usage task === [2024-09-26 16:01:08.453959] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 413,495,296 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 180,654,080 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 88,080,384 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4,194,304 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 12,582,912 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= RPC_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 17,969,152 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 11,292,672 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 92,430,336 limit= 9,223,372,036,854,775,807 [2024-09-26 16:01:08.454096] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:176) [2463][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 180,654,080 used= 173,078,944 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] hold= 33,574,912 used= 33,554,464 count= 1 avg_used= 33,554,464 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 18,894,848 used= 18,874,392 count= 1 avg_used= 18,874,392 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 12,460,032 used= 11,125,608 count= 169 avg_used= 65,832 block_cnt= 169 chunk_cnt= 29 mod=[T]ObSessionDIB [MEMORY] hold= 10,776,576 used= 10,735,904 count= 11 avg_used= 975,991 block_cnt= 11 chunk_cnt= 7 mod=IoControl [MEMORY] hold= 8,585,216 used= 8,566,048 count= 1 avg_used= 8,566,048 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 5,251,072 used= 5,238,784 count= 32 avg_used= 163,712 block_cnt= 32 chunk_cnt= 9 mod=Iteratorget_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6690, mem_used=12478464) [2024-09-26 16:01:08.566888] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=5] gc stale ls task succ [2024-09-26 16:01:08.575879] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:08.575888] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.586029] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337668361509]}) [2024-09-26 16:01:08.592580] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:08.592706] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:08.593351] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=723511, clean_num=31457) [2024-09-26 16:01:08.600802] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.602154] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.602189] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.602209] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.604804] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:01:08.605941] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:08.605950] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.617518] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=2] LogLoopThread round_cost_time(us)(round_cost_time=107) [2024-09-26 16:01:08.621908] INFO [SHARE.LOCATION] renew_all_ls_locations_by_rpc (ob_ls_location_service.cpp:783) [2618][AutoLSLocRpc][T0][YB42C0A8002D-0006230057E8589E-0-0] [lt=12] [LS_LOCATION] Get ls leaders by RPC(ret=0, ret="OB_SUCCESS", dests=["192.168.0.45:2882"], leaders=[{key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, location:{server:"192.168.0.45:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:374}}]) [2024-09-26 16:01:08.628291] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=8] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:08.628303] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:08.628308] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:08.628311] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:08.628315] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:08.628317] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:08.628321] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:08.628324] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:08.628326] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:08.628331] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=4][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:08.628333] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:08.628336] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:08.628338] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:08.628368] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=30][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:08.628376] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:08.628378] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:08.628382] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:08.628385] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:08.628388] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, ret=-5019) [2024-09-26 16:01:08.628391] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:08.628394] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:08.628400] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:08.628406] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:08.628408] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:08.628410] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:08.628416] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:08.628420] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=4][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:08.628423] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DD9-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-09-26 16:01:08.628426] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:01:08.628429] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:08.628439] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:08.628443] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337668628183, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:01:08.628447] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:08.628450] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f92b4056738, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, columns_str="zone") [2024-09-26 16:01:08.628457] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:01:08.628557] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:634) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f92b4056738, where_condition="where svr_ip='192.168.0.45' and svr_port=2882", zone_name_holder=) [2024-09-26 16:01:08.628562] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:567) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:01:08.628565] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:576) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:01:08.628567] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:08.628573] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:08.632974] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:08.632986] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337668632971) [2024-09-26 16:01:08.632991] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337668532905, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:08.633009] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223397869019, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:08.633015] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:08.633022] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:08.633028] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:08.633031] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:08.633037] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.633039] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.633041] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337668632994) [2024-09-26 16:01:08.636024] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:08.636032] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.665579] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:08.666171] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:08.666177] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.686375] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337668361509]}) [2024-09-26 16:01:08.696242] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:08.696252] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.726368] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:08.726377] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.733014] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:08.733028] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337668733010) [2024-09-26 16:01:08.733034] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337668632994, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:08.733039] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:08.733049] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:08.733053] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:08.733368] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:08.733378] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:08.733382] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:08.733386] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:08.733390] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:08.733392] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:08.733396] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:08.733398] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:08.733401] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:08.733403] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:08.733405] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:08.733408] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:08.733413] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=4][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:08.733415] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:08.733422] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:08.733424] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:08.733438] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:08.733441] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:08.733447] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=5][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:08.733450] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:08.733452] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:08.733478] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=23][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:08.733485] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:08.733487] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:08.733489] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:08.733495] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:08.733498] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:08.733502] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885787-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:08.733507] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:08.733510] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:08.733512] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:08.733515] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337668733294, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:08.733518] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:08.733522] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:08.733552] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:08.733555] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337668733555, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=510, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:08.733563] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:08.733566] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:08.733582] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223397969918, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:08.733589] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:08.733608] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:08.733614] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:08.733616] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:08.733622] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.733625] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.733626] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337668733569) [2024-09-26 16:01:08.756520] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:08.756550] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.761963] INFO [CLOG] get_replay_process (ob_replay_status.cpp:1073) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=3] replay status is not follower(min_unreplayed_lsn={lsn:629872191939}, base_lsn={lsn:627966418944}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:1, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:01:08.761988] INFO [CLOG] operator() (ob_log_replay_service.cpp:1590) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=25] get_replay_process success(id={id:1}, replayed_log_size=1905772995, unreplayed_log_size=0) [2024-09-26 16:01:08.762001] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:159) [2907][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=3] dump tenant replay process(tenant_id=1, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=1817, last_replayed_log_size(MB)=1817, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2024-09-26 16:01:08.786738] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337668361509]}) [2024-09-26 16:01:08.786760] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=79] success to notify_need_writing_throttling True [2024-09-26 16:01:08.786769] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.794682] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:08.794847] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=16] Wash time detail, (compute_wash_size_time=64, refresh_score_time=26, wash_time=2) [2024-09-26 16:01:08.795765] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=754968, clean_num=31457) [2024-09-26 16:01:08.805112] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:01:08.817022] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:08.817033] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.818494] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:08.821495] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.821509] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.821515] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.821715] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.821728] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.821734] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.821740] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.821744] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.821749] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.821819] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=1] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.821835] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:08.833089] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:08.833100] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337668833085) [2024-09-26 16:01:08.833105] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337668733037, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:08.833125] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223398069408, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:08.833131] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:08.833139] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:08.833144] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:08.833147] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:08.833152] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.833154] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.833156] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337668833109) [2024-09-26 16:01:08.847151] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:08.847235] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=82][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.861924] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:08.861945] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337668861922], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:08.861956] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337668861922], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:08.877356] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:08.877366] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.886908] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337668861922]}) [2024-09-26 16:01:08.907827] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:08.907837] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.922230] INFO [CLOG] run1 (ob_garbage_collector.cpp:1338) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=2] Garbage Collector is running(seq_=347, gc_interval=10000000) [2024-09-26 16:01:08.922250] INFO [CLOG] construct_server_ls_map_for_member_list_ (ob_garbage_collector.cpp:1473) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] self is leader, skip it(ls->get_ls_id()={id:1}) [2024-09-26 16:01:08.922258] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1431) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=4] gc_check_member_list_ cost time(ret=0, time_us=18) [2024-09-26 16:01:08.922262] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] execute_gc cost time(ret=0, time_us=0) [2024-09-26 16:01:08.922265] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1626) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=2] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2024-09-26 16:01:08.922269] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1648) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=3] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2024-09-26 16:01:08.922272] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1703) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=2] execute_gc cost time(ret=0, time_us=3) [2024-09-26 16:01:08.922274] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [2913][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=1] ObSafeDestroyHandler start process [2024-09-26 16:01:08.924546] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:250) [2910][T1_RFLWorker][T1][YB42C0A8002D-0006230055684B24-0-0] [lt=8] ObRemoteFetchWorker is running(thread_index=0) [2024-09-26 16:01:08.933248] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:08.933282] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337668933245) [2024-09-26 16:01:08.933288] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337668833109, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:08.933317] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223398169755, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:08.933323] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:08.933330] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:08.933335] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:08.933338] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:08.933343] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.933345] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:08.933347] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337668933291) [2024-09-26 16:01:08.938161] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:08.938168] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.949846] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1727337668949846, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000174, add_bytes: 0 [2024-09-26 16:01:08.965497] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-09-26 16:01:08.965508] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-09-26 16:01:08.965512] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-09-26 16:01:08.965515] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-09-26 16:01:08.965519] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:08.965522] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:08.965525] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-09-26 16:01:08.965530] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=4][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:08.965533] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:08.965535] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:08.965537] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:08.965539] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:08.965541] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:08.965543] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:08.965550] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:08.965552] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:08.965555] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:08.965557] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:08.965560] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:01:08.965563] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:08.965565] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:08.965571] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:08.965576] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:08.965578] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:08.965579] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:08.965585] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:08.965588] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:08.965591] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2024-09-26 16:01:08.965593] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:01:08.965596] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:08.965597] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:08.965600] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337668965385, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:01:08.965603] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:08.965606] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1111) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:01:08.965630] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:801) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:01:08.965635] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:67) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=5][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:08.965639] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:436) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:08.965641] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:95) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:08.965642] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:109) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=1] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:08.965866] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-09-26 16:01:08.965873] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2024-09-26 16:01:08.965877] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2024-09-26 16:01:08.965879] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2024-09-26 16:01:08.965881] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:08.965884] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:08.965886] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_tenant' doesn't exist [2024-09-26 16:01:08.965888] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:08.965889] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:08.965891] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:08.965892] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:08.965894] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:08.965895] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:08.965896] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:08.965899] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:08.965902] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:08.965904] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:08.965905] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:08.965907] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:01:08.965908] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:08.965909] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:08.965912] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:08.965915] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:08.965917] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:08.965918] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:08.965921] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:08.965923] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:08.965926] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, aret=-5019, ret=-5019) [2024-09-26 16:01:08.965928] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:01:08.965930] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:08.965932] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:08.965934] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337668965839, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:01:08.965936] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:08.965938] WDIAG [SHARE] read_tenants (ob_unit_table_operator.cpp:1174) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] execute sql failed(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:01:08.965950] WDIAG [SHARE] get_tenants (ob_unit_table_operator.cpp:108) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=3][errcode=-5019] fail read tenants(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:01:08.965953] WDIAG [SHARE] get_tenants (ob_unit_getter.cpp:198) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=2][errcode=-5019] ut_operator get_resource_pools failed(ret=-5019) [2024-09-26 16:01:08.965954] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:125) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=1][errcode=-5019] get cluster tenants fail(ret=-5019) [2024-09-26 16:01:08.965980] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:325) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=14] cal_all_part_disk_default_percentage succ(data_dir="/root/observer/store/sstable", clog_dir="/root/observer/store/clog", shared_mode=true, data_disk_total_size=99765694464, data_disk_default_percentage=60, clog_disk_total_size=99765694464, clog_disk_default_percentage=30) [2024-09-26 16:01:08.965985] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:361) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=4] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=99765694464, disk_size=15032385536) [2024-09-26 16:01:08.966003] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=18] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=99765694464) [2024-09-26 16:01:08.966007] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=3] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=99765694464, this={dir::"/root/observer/store/clog/log_pool", dir_fd:15, meta_fd:16, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:9388, max_block_id:9581, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2024-09-26 16:01:08.966013] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:133) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDA-0-0] [lt=5] refresh tenant config(tenants=[], ret=-5019) [2024-09-26 16:01:08.968434] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:08.968444] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:08.987182] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337668861922]}) [2024-09-26 16:01:08.992616] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:08.995918] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:08.996034] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=53, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:08.996671] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=2, clean_time=0, clean_start_pos=786425, clean_num=31457) [2024-09-26 16:01:08.998498] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:08.998527] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.005507] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=959408, replace_num=15728) [2024-09-26 16:01:09.005514] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:01:09.027176] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185942-0-0] [lt=45][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:09.027189] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185942-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:09.027198] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185942-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:09.027203] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185942-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:09.028737] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.028745] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.033343] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:09.033353] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337669033340) [2024-09-26 16:01:09.033357] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337668933291, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:09.033377] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223398268781, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:09.033383] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:09.033390] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:09.033394] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:09.033396] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:09.033402] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.033415] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.033417] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337669033361) [2024-09-26 16:01:09.050768] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500879, support_ssl=false) [2024-09-26 16:01:09.050806] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221500879, sess.client_addr_="127.0.0.1:36398") [2024-09-26 16:01:09.050813] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5] accept one succ(*s={this:0x7f9264604030, session_id:3221500879, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:09.050832] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:09.050843] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:09.050885] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA8-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:09.051105] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA8-0-0] [lt=6][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:09.051117] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA8-0-0] [lt=10][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"L**5j7r[P_l!\-jQ7^Sd"}, ret=-4043) [2024-09-26 16:01:09.051124] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA8-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:09.051128] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA8-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:09.051137] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA8-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:09.051159] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA8-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500879, proxy_sessid:0}) [2024-09-26 16:01:09.051163] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA8-0-0] [lt=4] mark session id unused(sessid=3221500879) [2024-09-26 16:01:09.051167] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA8-0-0] [lt=3][errcode=0] server close connection(sessid=3221500879, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:09.051171] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA8-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500879, proxy_sessid=0) [2024-09-26 16:01:09.051173] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA8-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:09.051176] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA8-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500879, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:09.051215] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f9264604030, session_id:3221500879, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337669050558, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:09.051229] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221500879, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:09.051242] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] can close safely, do destroy(*s={this:0x7f9264604030, session_id:3221500879, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337669050558, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:09.051248] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221500879, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:09.058769] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.058778] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.072265] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589E-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:09.072277] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589E-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:01:09.072281] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589E-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:01:09.072285] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589E-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:01:09.072288] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:09.072290] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:09.072294] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:01:09.072296] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589E-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:09.072298] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589E-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:09.072300] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589E-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:09.072302] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589E-0-0] [lt=1][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:09.072321] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589E-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:09.076946] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=11] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:09.077215] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7A-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:09.077224] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7A-0-0] [lt=8][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:09.077242] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7A-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:09.087523] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337668861922]}) [2024-09-26 16:01:09.089497] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:09.089538] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=40][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.092493] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=4] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:09.119654] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.119663] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.133461] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:09.133470] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:09.133477] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337669133458) [2024-09-26 16:01:09.133482] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337669033360, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:09.133499] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223398369097, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:09.133505] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:09.133512] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:09.133517] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:09.133532] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:09.133538] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.133540] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.133542] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337669133485) [2024-09-26 16:01:09.149733] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25] success to notify_need_writing_throttling True [2024-09-26 16:01:09.149741] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.156616] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-09-26 16:01:09.156945] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD9-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-09-26 16:01:09.156957] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD9-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:09.156982] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DD9-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:09.179769] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.179778] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.187655] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337668861922]}) [2024-09-26 16:01:09.189685] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 12088.00) [2024-09-26 16:01:09.189693] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:09.189705] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:09.189710] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:09.196909] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:09.197086] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=104, refresh_score_time=28, wash_time=1) [2024-09-26 16:01:09.197902] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=817882, clean_num=31457) [2024-09-26 16:01:09.205596] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:01:09.209829] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:09.209835] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.209844] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:09.213829] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:01:09.213841] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:09.213846] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:01:09.234072] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:09.234082] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337669234069) [2024-09-26 16:01:09.234097] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337669133484, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:09.234103] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:09.234111] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:09.234114] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:09.234429] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885788-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:09.234437] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885788-0-0] [lt=7][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:09.234463] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885788-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:09.234497] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337669234496, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=388, wlock_time=8, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:09.234505] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:09.234522] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223398470507, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:09.234527] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:09.234533] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:09.234538] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:09.234561] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:09.234567] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.234569] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.234571] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337669234509) [2024-09-26 16:01:09.243905] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:09.243912] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.265681] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:09.265693] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:09.265697] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:09.265700] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:09.265703] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:09.265705] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:09.265708] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:09.265710] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:09.265712] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:09.265714] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:09.265715] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:09.265717] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:09.265732] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:09.273970] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:09.273977] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.277104] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:161, tid:2919}]) [2024-09-26 16:01:09.288037] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337668861922]}) [2024-09-26 16:01:09.288044] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8D9F6-0-0] [lt=12][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337669288032}) [2024-09-26 16:01:09.304030] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:09.304037] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.326043] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337670255499, now=[mts=1727337668325781]) [2024-09-26 16:01:09.326053] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337670255499, now=[mts=1727337668325781], now0=[mts=1727337668325781], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:09.326060] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269029360, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337668325585, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:09.326079] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=18][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269029360}) [2024-09-26 16:01:09.326087] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269029360, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337668325585, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:09.326094] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269029360, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337668325585, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337668325750, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269029360}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:09.326111] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:09.326116] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:09.326118] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:09.326129] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=10][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:09.326132] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:09.326137] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:09.326141] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=999359, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337670325499) [2024-09-26 16:01:09.326144] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:09.326150] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:09.326152] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:09.326154] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:09.326158] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:09.326192] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:01:09.326195] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:09.334114] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:09.334121] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.334140] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:09.334147] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337669334138) [2024-09-26 16:01:09.334151] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337669234101, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:09.334167] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223398570457, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:09.334173] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:09.334179] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:09.334184] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:09.334187] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:09.334192] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.334194] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.334196] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337669334154) [2024-09-26 16:01:09.362614] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.362624] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337669362612], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:09.362630] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337669362612], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:09.362634] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:09.362637] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:09.364186] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:09.364193] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.388543] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337669362612]}) [2024-09-26 16:01:09.392678] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:213) [2916][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8] start to check and clear obj lock(ls->get_ls_id()={id:1}) [2024-09-26 16:01:09.392698] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=4] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:09.393518] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [2645][Blacklist][T0][Y0-0000000000000000-0-0] [lt=9] blacklist_loop exec finished(cost_time=9, is_enabled=true, send_cnt=0) [2024-09-26 16:01:09.394224] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.394232] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.398581] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:09.398683] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=51, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:09.399302] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=849339, clean_num=31457) [2024-09-26 16:01:09.406479] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:01:09.424288] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.424297] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.433016] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13] ====== tenant freeze timer task ====== [2024-09-26 16:01:09.433026] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:01:09.433057] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337665635081, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337665635046, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337665635102, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:01:09.433079] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:01:09.433177] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E7-0-0] [lt=3] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:01:09.433190] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E7-0-0] [lt=10] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:01:09.433196] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E7-0-0] [lt=4] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:01:09.433199] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E7-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:01:09.433204] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E7-0-0] [lt=4][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:01:09.433207] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E7-0-0] [lt=3][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:01:09.433210] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E7-0-0] [lt=3][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:01:09.433221] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E7-0-0] [lt=11] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:01:09.433225] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E7-0-0] [lt=4][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:09.433227] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E7-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:09.433230] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E7-0-0] [lt=2][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:09.433275] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851E7-0-0] [lt=6] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:01:09.434251] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:09.434264] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337669434248) [2024-09-26 16:01:09.434269] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337669334154, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:09.434285] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223398670085, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:09.434290] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:09.434297] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:09.434302] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:09.434305] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:09.434310] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.434312] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.434314] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337669434272) [2024-09-26 16:01:09.454428] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:09.454435] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.467596] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D4559-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:09.469685] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=11] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.469697] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.469703] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.470239] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.470243] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.470246] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.470853] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.470859] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.470861] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.471458] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.471462] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.471464] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.472070] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.472075] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.472078] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.472708] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.472711] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.472714] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.473290] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.473296] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.473299] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.473892] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.473895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.473897] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.474450] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.474460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.474462] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.475075] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.475080] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.475082] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.475589] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.475594] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.475597] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.476149] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.476154] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.476157] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.476673] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.476676] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.476678] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.477216] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.477219] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.477221] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.477797] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.477803] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.477805] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.478397] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.478402] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.478405] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:09.479016] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.479585] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=30][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.480140] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.480706] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.481269] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.481873] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.482462] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.483037] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.483589] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.484233] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.484518] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:09.484543] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.484787] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.485352] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.485894] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.486477] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.487061] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.487621] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.488170] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.488763] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.488775] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337669362612]}) [2024-09-26 16:01:09.489355] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.489924] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.490476] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.491073] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.491757] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.492361] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.492934] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.493479] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.494028] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.494683] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.495313] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.495862] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.496389] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.497078] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.497720] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.498834] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.499500] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.500203] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.500879] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.501463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.501837] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337669501837, bytes: 14739325, bw: 0.004029 MB/s, add_ts: 1007605, add_bytes: 4257 [2024-09-26 16:01:09.502061] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.502608] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.503200] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.503754] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.504279] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.504929] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.505513] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.506113] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.506705] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.507318] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.507873] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.508490] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.509123] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.509727] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.510373] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.510944] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.511456] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.512009] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.512643] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.513231] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.513880] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.514505] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.514655] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:09.514663] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.515103] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.515677] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.516227] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.516820] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.517384] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.517964] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.518478] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.519131] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.519727] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.520366] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.520977] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.521507] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.522154] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.522728] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.523328] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.523345] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185943-0-0] [lt=35][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:09.523356] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185943-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:09.523365] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185943-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:09.523371] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185943-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:09.523934] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.524524] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.525183] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=42][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.525721] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.526241] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.526823] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.527348] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.527929] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.528465] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:09.534351] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:09.534362] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337669534348) [2024-09-26 16:01:09.534367] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337669434272, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:09.534388] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223398770117, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:09.534394] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:09.534402] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:09.534407] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:09.534409] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:09.534416] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.534418] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.534419] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337669534370) [2024-09-26 16:01:09.544705] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.544715] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.551474] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6700, mem_used=12478464) [2024-09-26 16:01:09.566977] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:312) [2940][T1_ArcFetcher][T1][YB42C0A8002D-0006230055E84B24-0-0] [lt=6] ObArchiveFetcher is running(thread_index=0) [2024-09-26 16:01:09.567498] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=6] gc stale ls task succ [2024-09-26 16:01:09.574831] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:09.574839] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.589302] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337669362612]}) [2024-09-26 16:01:09.599536] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:09.599646] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash time detail, (compute_wash_size_time=104, refresh_score_time=17, wash_time=2) [2024-09-26 16:01:09.600287] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=880796, clean_num=31457) [2024-09-26 16:01:09.601481] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.602161] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=7] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.602191] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.602198] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.604933] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.604967] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=29][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.607459] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:01:09.629448] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:09.634383] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223398869998, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:09.634393] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:09.634403] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:09.634410] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:09.634413] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:09.634419] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.634421] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.634423] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337669634367) [2024-09-26 16:01:09.635015] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:09.635021] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.665111] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.665120] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.666489] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:09.669844] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=18] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:01:09.672325] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855FB-0-0] [lt=7] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:09.689738] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337669362612]}) [2024-09-26 16:01:09.695220] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.695229] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.725284] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:09.725294] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.734435] WDIAG [PALF] convert_to_ts (scn.cpp:265) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2024-09-26 16:01:09.734443] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1727337669734433, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1727337669734433, force_self_check=false, weak_read_refresh_interval=100000) [2024-09-26 16:01:09.734471] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:09.734477] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337669734469) [2024-09-26 16:01:09.734481] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337669534370, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:09.734496] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223398970203, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:09.734503] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:09.734510] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:09.734515] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:09.734518] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:09.734524] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.734527] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.734529] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337669734483) [2024-09-26 16:01:09.739674] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=9] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:01:09.755358] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.755366] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.761754] INFO [STORAGE.TRANS] print_statistics (ob_xa_define.cpp:467) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=8] xa statistics(total_active_xa_ctx_count=0, total_standby_clearup_count=0, total_success_xa_start=0, total_failure_xa_start=0, total_success_xa_prepare=0, total_failure_xa_prepare=0, total_success_xa_1pc_commit=0, total_failure_xa_1pc_commit=0, total_success_xa_2pc_commit=0, total_failure_xa_2pc_commit=0, total_failure_xa_rollback=0, total_success_dblink_promotion=0, total_failure_dblink_promotion=0, total_success_dblink=0, total_failure_dblink=0) [2024-09-26 16:01:09.761839] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=84] XA scheduler heartbeat task statistics(avg_time=85) [2024-09-26 16:01:09.784910] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=3] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-09-26 16:01:09.784919] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-09-26 16:01:09.784924] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=4] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=694) [2024-09-26 16:01:09.785878] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.785887] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.789801] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337669362612]}) [2024-09-26 16:01:09.800543] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:09.800645] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash time detail, (compute_wash_size_time=62, refresh_score_time=16, wash_time=1) [2024-09-26 16:01:09.801239] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=912253, clean_num=31457) [2024-09-26 16:01:09.808491] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:01:09.815919] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.815929] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.819474] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:09.821281] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=7] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.821297] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.821304] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.821498] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.821511] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.821517] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.821523] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.821528] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.821532] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.821536] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=10] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.821544] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:09.834470] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:09.834484] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:09.834488] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:09.834830] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:09.834842] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:09.834847] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:09.834853] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=6][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:09.834857] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:09.834859] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:09.834863] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:09.834865] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:09.834868] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:09.834870] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:09.834872] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:09.834881] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:09.834884] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:09.834886] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:09.834892] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:09.834895] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:09.834898] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:09.834902] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:09.834905] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:09.834908] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:09.834910] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:09.834917] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:09.834923] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:09.834925] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:09.834929] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:09.834936] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=3][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:09.834939] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:09.834942] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885789-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:09.834945] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:09.834947] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:09.834952] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:09.834954] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337669834754, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:09.834958] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:09.834960] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:09.834990] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:09.834993] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337669834992, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=512, wlock_time=10, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:09.835001] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:09.835004] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:09.835017] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get_max_applied_scn(scn={val:1727114270763524000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:374, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:1727114270763524000, v:0}, max_applied_cb_scn_:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:09.835026] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] get_min_unreplayed_log_info(lsn={lsn:629872191939}, scn={val:1727114270763524001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:01:09.835044] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=223399070940, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:09.835049] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:09.835055] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:09.835059] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:09.835062] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:09.835068] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.835070] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.835072] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337669835008) [2024-09-26 16:01:09.846072] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.846080] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.863298] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:09.863310] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337669863295], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:09.863318] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337669863295], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:09.876231] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:09.876241] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.880938] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=13] table not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, table_name.ptr()="data_size:16, data:5F5F616C6C5F6D657267655F696E666F", ret=-5019) [2024-09-26 16:01:09.880962] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=22][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, ret=-5019) [2024-09-26 16:01:09.880967] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_merge_info, db_name=oceanbase) [2024-09-26 16:01:09.880970] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_merge_info) [2024-09-26 16:01:09.880974] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:09.880976] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:09.880979] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_merge_info' doesn't exist [2024-09-26 16:01:09.880982] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:09.880984] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:09.880986] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:09.880987] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:09.880990] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:09.880992] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:09.880994] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:09.881000] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:09.881002] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:09.881005] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:09.881007] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:09.881012] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_merge_info WHERE tenant_id = '1', ret=-5019) [2024-09-26 16:01:09.881015] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:09.881017] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:09.881023] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:09.881029] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:09.881031] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:09.881064] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:09.881074] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=5][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:09.881078] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:09.881081] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508E-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, aret=-5019, ret=-5019) [2024-09-26 16:01:09.881083] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:01:09.881086] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:09.881089] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:09.881094] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337669880875, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:01:09.881098] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:09.881100] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:49) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:01:09.881130] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:890) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] failed to load global merge info(ret=-5019, ret="OB_TABLE_NOT_EXIST", global_merge_info={tenant_id:1, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2024-09-26 16:01:09.881142] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1005) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] fail to refresh merge info(tmp_ret=-5019, tmp_ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:09.881375] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:01:09.881384] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=8][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:09.890050] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337669863295]}) [2024-09-26 16:01:09.906449] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:09.906462] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.921816] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] ====== checkpoint timer task ====== [2024-09-26 16:01:09.922488] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] succ to get rec scn(*this={this:0x7f92a6a04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, total_active_readonly_request_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1726077600526835000, v:0}, last_push_gc_task_ts_:1727334198745636, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:01:09.922608] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:182) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=119] tx ctx memtable get rec scn(this={ObITable:{this:0x7f92a22a4080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f92a22a4080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2024-09-26 16:01:09.922623] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:790) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:1723174350926847000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2024-09-26 16:01:09.922641] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:260) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:01:09.922682] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:524) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=38] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}, min_rec_scn={val:1726163586617842000, v:0}, ls_id_={id:1}) [2024-09-26 16:01:09.922747] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=10] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:01:09.922753] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:01:09.922757] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=2] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2024-09-26 16:01:09.922763] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:180) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=1] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1726163586617842000, v:0}, checkpoint_scn_in_ls_meta={val:1726163586617842000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2024-09-26 16:01:09.922789] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1554) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"192.168.0.45:2882", has_set_deleted:false}, curr_base_lsn={lsn:627966418944}, new_base_lsn={lsn:627966418944}, lsn={lsn:627966418944}) [2024-09-26 16:01:09.922796] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:627966418944}, ls->get_ls_id()={id:1}) [2024-09-26 16:01:09.922800] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [2933][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=3] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2024-09-26 16:01:09.934614] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:09.934645] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337669934609) [2024-09-26 16:01:09.934651] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337669734483, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:09.934670] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223399168605, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:09.934677] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:09.934685] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:09.934691] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:09.934694] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:09.934700] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.934703] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:09.934705] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337669934654) [2024-09-26 16:01:09.936529] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:01:09.936536] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.956625] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1727337669956625, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006779, add_bytes: 0 [2024-09-26 16:01:09.966584] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:09.966593] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:09.980386] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [2708][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=165] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"2GB", log_disk_size:"2GB", min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3}}, mode:0, create_timestamp:1710828706488537, is_removed:false}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=15, log_id=1828900, offset=14449774}, ls_meta_entry:[10007](ver=0,mode=0,seq=609892), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:false, version:2}, create_status:1}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:49, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:80737413, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:1740, recv_lp_rpc_cnt:0, recv_mysql_cnt:1154, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:9793976, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:group_id = 2,queue_size = 0,recv_req_cnt = 7624,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334211521784 group_id = 10,queue_size = 0,recv_req_cnt = 3452,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334216415596 group_id = 9,queue_size = 0,recv_req_cnt = 7056,min_worker_cnt = 3,max_worker_cnt = 150,worker_cnt = 3,token_change = 1727334199387293 group_id = 1,queue_size = 0,recv_req_cnt = 4466,min_worker_cnt = 8,max_worker_cnt = 8,worker_cnt = 0,token_change = 1727336105580026 , rpc_stat_info: pcode=0x150a:cnt=20 pcode=0x150b:cnt=20 pcode=0x4a9:cnt=10 pcode=0x113:cnt=5, token_change_ts:1727334199363389, tenant_role:1}) [2024-09-26 16:01:09.980604] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2335) [2708][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=217] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:1, max_cpu:1, memory_size:"1GB", log_disk_size:"2GB", min_iops:10000, max_iops:50000, iops_weight:0}}, mode:0, create_timestamp:1727334198122806, is_removed:false}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:[-1](ver=0,mode=0,seq=0), tablet_meta_entry:[-1](ver=0,mode=0,seq=0), is_hidden:true, version:2}, create_status:1}, unit_min_cpu:"1.000000000000000000e+00", unit_max_cpu:"1.000000000000000000e+00", total_worker_cnt:12, min_worker_cnt:12, max_worker_cnt:150, stopped:0, worker_us:2121351, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, pop_normal_cnt:3264385, workers:12, nesting workers:0, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1727334198125585, tenant_role:0}) [2024-09-26 16:01:09.990470] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337669863295]}) [2024-09-26 16:01:09.992082] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1916) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=4] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2024-09-26 16:01:09.992091] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1457) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=7] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:01:09.992096] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1474) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=4] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=107374180, cache_obj_num=3, cache_node_num=3) [2024-09-26 16:01:09.992099] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2668) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:01:09.994007] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2387) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=2] Dumping All Cache Objs(alloc_obj_list.count()=3, alloc_obj_list=[{obj_id:78, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:157779}, {obj_id:79, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:1728}, {obj_id:199, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:4, added_to_lc:true, mem_used:175172}]) [2024-09-26 16:01:09.994021] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2676) [2837][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=12] schedule next cache evict task(evict_interval=5000000) [2024-09-26 16:01:09.994495] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:09.996646] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:09.996655] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.001495] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:10.001604] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=80, refresh_score_time=19, wash_time=2) [2024-09-26 16:01:10.002216] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=943710, clean_num=31457) [2024-09-26 16:01:10.008570] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:01:10.020887] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185944-0-0] [lt=38][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:10.020897] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185944-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:10.020905] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185944-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:10.020911] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185944-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:10.026848] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21] success to notify_need_writing_throttling True [2024-09-26 16:01:10.026858] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.034687] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:10.034700] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337670034683) [2024-09-26 16:01:10.034706] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337669934654, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:10.034724] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223399270671, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.034730] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:10.034738] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:10.034743] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:10.034746] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:10.034753] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.034756] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.034758] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337670034708) [2024-09-26 16:01:10.052540] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] new mysql sessid created(conn.sessid_=3221500886, support_ssl=false) [2024-09-26 16:01:10.052559] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221500886, sess.client_addr_="127.0.0.1:36400") [2024-09-26 16:01:10.052565] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f9262604030, session_id:3221500886, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:10.052609] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:10.052618] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:10.052659] INFO load_privilege_info (obmp_connect.cpp:532) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA8-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:10.052881] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA8-0-0] [lt=8][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:10.052920] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA8-0-0] [lt=37][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"MzBxW+lc+Kv26V0-.@89"}, ret=-4043) [2024-09-26 16:01:10.052928] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA8-0-0] [lt=6][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:10.052932] WDIAG [SERVER] process (obmp_connect.cpp:340) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA8-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:10.052941] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA8-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:10.052972] INFO [SERVER] free_session (obmp_base.cpp:324) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA8-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500886, proxy_sessid:0}) [2024-09-26 16:01:10.052977] INFO [SERVER] free_session (obmp_base.cpp:332) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA8-0-0] [lt=5] mark session id unused(sessid=3221500886) [2024-09-26 16:01:10.052982] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA8-0-0] [lt=3][errcode=0] server close connection(sessid=3221500886, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:10.052985] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA8-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500886, proxy_sessid=0) [2024-09-26 16:01:10.052988] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA8-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:10.052993] INFO [SERVER] process (obmp_connect.cpp:475) [2923][T1_MysqlQueueTh][T1][Y0-0006230056784FA8-0-0] [lt=5] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500886, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:10.053021] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f9262604030, session_id:3221500886, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337670051726, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:10.053039] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221500886, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:10.053043] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f9262604030, session_id:3221500886, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337670051726, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:10.053047] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500886, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:10.057315] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:10.057325] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.072694] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:10.072705] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:01:10.072710] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:01:10.072713] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:01:10.072717] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:10.072720] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:10.072724] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:01:10.072727] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:10.072729] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:10.072731] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:10.072734] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:10.072737] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:10.072739] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:10.072744] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=5][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:10.072751] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:10.072753] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:10.072756] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:10.072759] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:10.072761] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:01:10.072764] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:10.072766] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:10.072772] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:10.072778] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:10.072780] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:10.072784] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=4][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:10.072791] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:10.072794] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:10.072798] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589F-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:01:10.072801] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:10.072804] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:10.072806] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:10.072809] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337670072615, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:10.072812] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589F-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:10.072815] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:01:10.072838] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:01:10.072840] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C8589F-0-0] [lt=2][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:01:10.077367] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=7] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:10.077783] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:10.077794] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=10][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:10.077798] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:10.077801] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:10.077804] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:10.077807] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:10.077810] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:10.077812] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:10.077817] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=5][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:10.077819] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:10.077821] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:10.077823] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:10.077825] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:10.077839] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=14][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:10.077845] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:10.077848] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:10.077851] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:10.077853] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:10.077858] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=4][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:01:10.077861] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:10.077863] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:10.077877] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=13][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:10.077883] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:10.077885] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:10.077887] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:10.077893] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:10.077896] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:10.077899] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2579][RSAsyncTask0][T0][YB42C0A8002D-0006230057584E7C-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:01:10.077902] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:10.077905] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:10.077909] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:10.077941] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337670077706, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:10.077945] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:10.077992] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:01:10.087372] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:10.087383] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.090717] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337669863295]}) [2024-09-26 16:01:10.092529] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:10.118003] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:10.118018] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.134778] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:10.134788] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:10.134794] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337670134775) [2024-09-26 16:01:10.134799] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337670034708, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:10.134816] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223399370692, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.134823] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:10.134829] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:10.134835] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:10.134837] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:10.134843] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.134845] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.134848] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337670134802) [2024-09-26 16:01:10.148093] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:10.148100] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.178249] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:10.178258] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.191186] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 9287.00) [2024-09-26 16:01:10.191196] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:10.191209] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:10.191213] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:10.191276] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337669863295]}) [2024-09-26 16:01:10.202422] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:10.202549] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] Wash time detail, (compute_wash_size_time=60, refresh_score_time=16, wash_time=1) [2024-09-26 16:01:10.203179] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=975167, clean_num=31457) [2024-09-26 16:01:10.204484] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] ====== check clog disk timer task ====== [2024-09-26 16:01:10.204497] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=10] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:01:10.204507] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:01:10.208354] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:10.208363] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.208657] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:01:10.234952] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:10.234965] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337670234949) [2024-09-26 16:01:10.234971] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337670134802, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:10.234989] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223399470910, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.234997] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:10.235004] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:10.235009] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:10.235012] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:10.235018] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.235021] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.235023] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337670234975) [2024-09-26 16:01:10.238584] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:10.238593] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.238605] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:10.266211] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:10.266222] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:10.266226] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:10.266229] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:10.266233] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:10.266235] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:10.266239] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:10.266241] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:10.266243] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:10.266244] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:10.266247] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:10.266251] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:10.266253] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:10.266256] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:10.266262] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:10.266280] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=18][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:10.266284] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:10.266287] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:10.266289] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:01:10.266292] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:10.266294] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:10.266300] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:10.266305] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:10.266308] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:10.266309] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:10.266315] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:10.266318] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:10.266320] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:01:10.266323] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:10.266326] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:10.266328] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:10.266330] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8060, start=1727337670266138, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:10.266333] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:10.266336] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:10.266365] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:10.266368] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:01:10.266379] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=11][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:10.266382] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:10.266385] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:10.266388] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:10.266389] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AE-0-0] [lt=1][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:10.268738] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:01:10.268747] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.291278] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337669863295]}) [2024-09-26 16:01:10.291306] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8DA02-0-0] [lt=17][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337670291270}) [2024-09-26 16:01:10.298779] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:10.298787] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.327950] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337670255499, now=[mts=1727337669326191]) [2024-09-26 16:01:10.327961] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=10][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337670255499, now=[mts=1727337669326191], now0=[mts=1727337669326191], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:10.327970] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f9269029360, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337669326040, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:10.327991] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=20][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269029360}) [2024-09-26 16:01:10.328000] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=9][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f9269029360, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337669326040, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:10.328011] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=10] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f9269029360, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337669326040, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337668325750, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f9269029360}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:10.328048] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=36][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:10.328055] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:10.328058] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:10.328061] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:10.328064] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:10.328068] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:10.328073] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:01:10.328082] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=6][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:10.328085] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:10.328087] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:01:10.328093] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:01:10.328098] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=3] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2002347) [2024-09-26 16:01:10.328101] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:01:10.328106] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:10.328110] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:01:10.328112] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:01:10.328116] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:01:10.328122] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:10.328157] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:01:10.328161] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:01:10.328164] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:01:10.328167] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:10.328172] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=4][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:10.328176] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=3][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:01:10.328294] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE4-0-0] [lt=5][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:01:10.328301] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:01:10.328304] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:01:10.328323] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=5] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:10.328340] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:01:10.328651] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:01:10.328657] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:10.328911] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:10.328919] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.335596] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:10.335606] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337670335593) [2024-09-26 16:01:10.335610] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337670234975, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:10.335615] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:10.335624] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:10.335626] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:10.335958] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:10.335967] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=7][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:10.335970] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:10.335974] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:10.335977] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:10.335980] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:10.335984] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:10.335986] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:10.335989] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:10.335991] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:10.335994] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:10.335996] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:10.336002] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=5][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:10.336004] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:10.336010] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:10.336013] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:10.336016] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:10.336019] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:10.336021] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:10.336024] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:10.336026] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:10.336031] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:10.336036] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=5][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.336044] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578A-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:10.336070] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337670336070, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=450, wlock_time=18, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:10.336078] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:10.336105] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=223399572099, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.336111] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:10.336117] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:10.336122] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:10.336124] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:10.336140] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.336143] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.336145] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337670336095) [2024-09-26 16:01:10.359294] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:10.359302] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.363682] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.363694] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337670363680], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:10.363701] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337670363680], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:10.363742] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:10.363746] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:10.389536] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:10.389544] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.391438] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337670363680]}) [2024-09-26 16:01:10.393204] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=11] current monitor number(seq_=-1) [2024-09-26 16:01:10.393848] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=4] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:10.403302] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:10.403403] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash time detail, (compute_wash_size_time=49, refresh_score_time=14, wash_time=1) [2024-09-26 16:01:10.404036] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1006624, clean_num=31457) [2024-09-26 16:01:10.409033] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:01:10.419825] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:10.419833] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.434463] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=9] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService13ObRestartTaskE) [2024-09-26 16:01:10.434488] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:366) [2582][RSAsyncTask3][T0][YB42C0A8002D-0006230057684E63-0-0] [lt=2] [RS_MGR] new master rootserver found(rootservice="192.168.0.45:2882", cluster_id=1) [2024-09-26 16:01:10.434507] INFO [SHARE.PT] update_leader_replica_ (ob_inmemory_ls_table.cpp:230) [2582][RSAsyncTask3][T0][YB42C0A8002D-0006230057684E63-0-0] [lt=4] add leader replica for sys ls(new_replica={modify_time_us:1727337670434507, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"192.168.0.45:2882", sql_port:2881, role:1, member_list:[{server:"192.168.0.45:2882", timestamp:1}], replica_type:0, proposal_id:374, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, ls_info_={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:1727337670434507, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"192.168.0.45:2882", sql_port:2881, role:1, member_list:[{server:"192.168.0.45:2882", timestamp:1}], replica_type:0, proposal_id:374, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-09-26 16:01:10.434533] INFO [SHARE.PT] update (ob_inmemory_ls_table.cpp:145) [2582][RSAsyncTask3][T0][YB42C0A8002D-0006230057684E63-0-0] [lt=3] update ls replica(ret=0, ret="OB_SUCCESS", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"192.168.0.45:2882", sql_port:2881, role:1, member_list:[{server:"192.168.0.45:2882", timestamp:1}], replica_type:0, proposal_id:374, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[{modify_time_us:1727337670434507, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"192.168.0.45:2882", sql_port:2881, role:1, member_list:[{server:"192.168.0.45:2882", timestamp:1}], replica_type:0, proposal_id:374, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:true, member_time_us:1, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}]}) [2024-09-26 16:01:10.434558] INFO [SHARE] store (ob_rs_mgr.cpp:128) [2582][RSAsyncTask3][T0][YB42C0A8002D-0006230057684E63-0-0] [lt=2] store rs list succeed(agent=0, addr_list=[{server:"192.168.0.45:2882", role:1, sql_port:2881}], force=true) [2024-09-26 16:01:10.434789] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F57-0-0] [lt=3] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:01:10.434800] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F57-0-0] [lt=6] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:10.434806] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F57-0-0] [lt=5][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.434812] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F57-0-0] [lt=0] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:01:10.434838] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F57-0-0] [lt=0] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:10.434845] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=74) [2024-09-26 16:01:10.434850] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:10.436052] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:10.436066] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337670436050) [2024-09-26 16:01:10.436071] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337670335613, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:10.436088] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223399672245, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.436094] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:10.436101] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:10.436115] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:10.436121] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:10.436127] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.436129] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.436131] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337670436076) [2024-09-26 16:01:10.449922] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] success to notify_need_writing_throttling True [2024-09-26 16:01:10.449928] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.467632] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D45BB-0-0] [lt=18][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:10.469618] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=10] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.469618] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.469625] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.469624] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.469629] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.469629] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.469638] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.469641] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.469642] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.470220] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.470220] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.470223] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.470225] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.470225] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.470227] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.470233] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.470234] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.470236] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.470753] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.470756] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.470756] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.470758] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.470758] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.470760] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.470767] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.470769] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.470771] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.471323] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.471328] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.471331] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.471357] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.471360] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.471362] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.471367] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.471369] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.471370] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.471901] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.471902] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.471905] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.471907] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.471912] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.471914] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.471906] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.471915] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.471917] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=10][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.472507] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.472511] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.472512] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.472514] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:10.472519] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.473106] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.473266] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.473272] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.473668] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.473846] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.473855] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.474282] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.474493] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.474520] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.474887] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.475136] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.475143] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.475445] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.475698] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.475708] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.476183] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.476406] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.476424] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.476753] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.477002] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.477009] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.477347] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.477587] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.477595] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.477925] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.478187] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.478200] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.478487] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.478714] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.478718] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.479063] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.479276] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.479281] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.479666] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.479878] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.479883] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.480089] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:10.480094] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.480446] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.480450] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.480460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.481313] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.481320] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.481321] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.481914] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.481922] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.481925] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.482454] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.482460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.482464] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.483021] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.483056] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.483061] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.483557] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.483617] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.483622] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.484142] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.484169] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.484198] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.484744] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.484750] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=26][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.484754] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.485269] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.485274] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.485305] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.485868] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.485895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.485901] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.486477] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=25][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.486483] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.486508] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.487062] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.487068] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.487100] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.487739] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.487743] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.487744] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.488314] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.488315] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.488320] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.488895] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.488946] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.488950] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.489479] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:10.491528] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337670363680]}) [2024-09-26 16:01:10.509447] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337670509446, bytes: 14742430, bw: 0.002939 MB/s, add_ts: 1007609, add_bytes: 3105 [2024-09-26 16:01:10.510268] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:01:10.510277] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.520577] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185945-0-0] [lt=41][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:10.520587] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185945-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:10.520596] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185945-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:10.520601] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185945-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:10.536155] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:10.536168] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337670536152) [2024-09-26 16:01:10.536173] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337670436075, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:10.536191] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223399771773, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.536197] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:10.536205] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:10.536209] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:10.536212] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:10.536219] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.536222] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.536223] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337670536176) [2024-09-26 16:01:10.540429] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:10.540436] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.551529] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6708, mem_used=12478464) [2024-09-26 16:01:10.567576] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=6] gc stale ls task succ [2024-09-26 16:01:10.570481] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:10.570488] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.591724] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337670363680]}) [2024-09-26 16:01:10.600554] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:10.600563] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.602185] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.602197] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.602204] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.602209] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=8] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.604207] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:10.604319] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=52, refresh_score_time=14, wash_time=3) [2024-09-26 16:01:10.604939] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=1, clean_time=0, clean_start_pos=1038081, clean_num=31457) [2024-09-26 16:01:10.609190] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:01:10.630330] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:10.630647] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:10.630657] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.636103] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] ====== traversal_flush timer task ====== [2024-09-26 16:01:10.636133] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c3a030, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6163825430799493735, dag_cnt=1, dag_type_cnts=1) [2024-09-26 16:01:10.636141] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] success to create and add dag(ret=0, dag=0x7f9264c3a030) [2024-09-26 16:01:10.636142] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1961) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(dag_cnt=1, map_size=1) [2024-09-26 16:01:10.636148] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1971) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(running_dag_net_map_size=0, blocking_dag_net_list_size=0) [2024-09-26 16:01:10.636151] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(priority="PRIO_COMPACTION_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=1, waiting_dag_count=0) [2024-09-26 16:01:10.636155] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(priority="PRIO_HA_HIGH", low_limit=8, up_limit=8, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:10.636159] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(priority="PRIO_COMPACTION_MID", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:10.636161] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_HA_MID", low_limit=5, up_limit=5, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:10.636145] INFO [STORAGE] flush (ob_tx_data_memtable.cpp:917) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] [TX DATA MERGE]schedule flush tx data memtable task done(ret=0, ret="OB_SUCCESS", get_ls_id()={id:1}, this=0x7f92a2294080, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:4, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337670636144, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:01:10.636163] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_COMPACTION_LOW", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:10.636165] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_HA_LOW", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:10.636166] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(priority="PRIO_DDL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:10.636168] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(priority="PRIO_DDL_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:10.636169] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(priority="PRIO_TTL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2024-09-26 16:01:10.636171] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:01:10.636175] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(type={init_dag_prio:0, sys_task_type:3, dag_type_str:"MINI_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=182) [2024-09-26 16:01:10.636177] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:01:10.636179] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:2, sys_task_type:5, dag_type_str:"MINOR_EXECUTE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:01:10.636180] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:01:10.636183] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:4, sys_task_type:6, dag_type_str:"MAJOR_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:01:10.636184] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=1) [2024-09-26 16:01:10.636186] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:4, dag_type_str:"TX_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=2) [2024-09-26 16:01:10.636187] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:01:10.636189] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:4, sys_task_type:7, dag_type_str:"WRITE_CKPT", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:01:10.636191] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, dag_count=0) [2024-09-26 16:01:10.636193] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:0, sys_task_type:19, dag_type_str:"MDS_TABLE_MERGE", dag_module_str:"COMPACTION"}, scheduled_task_count=0) [2024-09-26 16:01:10.636194] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:01:10.636196] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"DDL", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:01:10.636199] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:01:10.636181] INFO [MDS] flush (mds_table_mgr.cpp:172) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=19] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f92a6c04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2024-09-26 16:01:10.636201] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"UNIQUE_CHECK", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:01:10.636203] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:01:10.636205] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:6, sys_task_type:2, dag_type_str:"SQL_BUILD_INDEX", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:01:10.636206] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, dag_count=0) [2024-09-26 16:01:10.636207] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:7, sys_task_type:12, dag_type_str:"DDL_KV_MERGE", dag_module_str:"DDL"}, scheduled_task_count=0) [2024-09-26 16:01:10.636209] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636211] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636213] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636216] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636217] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636217] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=21] add dag success(dag=0x7f9264c3a560, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6659188001345271846, dag_cnt=2, dag_type_cnts=1) [2024-09-26 16:01:10.636219] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_COMPLETE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636221] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636221] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c3a560) [2024-09-26 16:01:10.636224] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636226] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636228] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636241] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636224] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, create_flush_dag_time:1727337670636224, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452867, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200240}, checkpoint_unit_ptr:0x7f92a1a70850}, is_freeze_clock:true}) [2024-09-26 16:01:10.636246] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"SYS_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636248] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636249] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636251] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636252] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"DATA_TABLETS_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636254] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636255] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"TABLET_GROUP_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636245] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=20] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337670636210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636257] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636258] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"MIGRATION_FINISH", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636260] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636261] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"INITIAL_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636263] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636263] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=14] add dag success(dag=0x7f9264c3aa90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-9069613569758001279, dag_cnt=3, dag_type_cnts=2) [2024-09-26 16:01:10.636265] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"START_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636265] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3aa90) [2024-09-26 16:01:10.636266] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636268] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:1, dag_type_str:"FINISH_PREPARE_MIGRATION", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636267] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, create_flush_dag_time:1727337670636267, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452892, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:200489}, checkpoint_unit_ptr:0x7f92a1a70080}, is_freeze_clock:true}) [2024-09-26 16:01:10.636270] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636271] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:3, sys_task_type:1, dag_type_str:"FAST_MIGRATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636273] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, dag_count=0) [2024-09-26 16:01:10.636275] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:1, dag_type_str:"VALIDATE", dag_module_str:"MIGRATE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636276] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-09-26 16:01:10.636271] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337670636259, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636279] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"TABLET_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-09-26 16:01:10.636280] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, dag_count=0) [2024-09-26 16:01:10.636281] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3afc0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1288986994123435108, dag_cnt=4, dag_type_cnts=3) [2024-09-26 16:01:10.636282] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264c3afc0) [2024-09-26 16:01:10.636284] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:16, dag_type_str:"FINISH_BACKFILL_TX", dag_module_str:"BACKFILL_TX"}, scheduled_task_count=0) [2024-09-26 16:01:10.636286] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:10.636284] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337670636284, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452911, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847222}, checkpoint_unit_ptr:0x7f92a1a71020}, is_freeze_clock:true}) [2024-09-26 16:01:10.636287] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_META", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:10.636289] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:10.636290] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_PREPARE", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:10.636292] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:10.636288] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337670636278, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636293] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_FINISH", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:10.636295] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:10.636296] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_DATA", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:10.636296] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c3b4f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5355160120942094702, dag_cnt=5, dag_type_cnts=4) [2024-09-26 16:01:10.636297] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:10.636298] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3b4f0) [2024-09-26 16:01:10.636299] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"PREFETCH_BACKUP_INFO", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:10.636299] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, create_flush_dag_time:1727337670636299, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334201452929, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:0, key:{tablet_id:{id:1152921504606847223}, checkpoint_unit_ptr:0x7f92a1a717f0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636301] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:10.636303] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_INDEX_REBUILD", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:10.636305] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:10.636306] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:8, dag_type_str:"BACKUP_COMPLEMENT_LOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:10.636308] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:10.636309] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:10, dag_type_str:"BACKUP_BACKUPSET", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:10.636311] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, dag_count=0) [2024-09-26 16:01:10.636312] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:11, dag_type_str:"BACKUP_ARCHIVELOG", dag_module_str:"BACKUP"}, scheduled_task_count=0) [2024-09-26 16:01:10.636313] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:10.636315] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636316] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:10.636318] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636321] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:10.636305] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337670636294, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636322] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"SYS_TABLETS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636324] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:10.636325] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"DATA_TABLETS_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636328] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:10.636330] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_GROUP_META_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636332] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:10.636333] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_LS_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636328] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405042) [2024-09-26 16:01:10.636335] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:10.636336] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"INITIAL_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636337] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:10.636339] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"START_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636339] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:01:10.636340] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:10.636342] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"FINISH_TABLET_GROUP_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636343] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, dag_count=0) [2024-09-26 16:01:10.636345] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:14, dag_type_str:"TABLET_RESTORE", dag_module_str:"RESTORE"}, scheduled_task_count=0) [2024-09-26 16:01:10.636346] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, dag_count=0) [2024-09-26 16:01:10.636348] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:5, sys_task_type:15, dag_type_str:"BACKUP_CLEAN", dag_module_str:"BACKUP_CLEAN"}, scheduled_task_count=0) [2024-09-26 16:01:10.636349] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, dag_count=0) [2024-09-26 16:01:10.636351] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:3, sys_task_type:17, dag_type_str:"REMOVE_MEMBER", dag_module_str:"REMOVE_MEMBER"}, scheduled_task_count=0) [2024-09-26 16:01:10.636346] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405070) [2024-09-26 16:01:10.636352] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, dag_count=0) [2024-09-26 16:01:10.636354] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_BACKFILL_TX", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-09-26 16:01:10.636354] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:01:10.636355] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, dag_count=0) [2024-09-26 16:01:10.636358] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:1, sys_task_type:18, dag_type_str:"TRANSFER_REPLACE_TABLE", dag_module_str:"TRANSFER"}, scheduled_task_count=0) [2024-09-26 16:01:10.636360] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1983) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, dag_count=0) [2024-09-26 16:01:10.636357] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405087) [2024-09-26 16:01:10.636362] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1984) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status(type={init_dag_prio:8, sys_task_type:20, dag_type_str:"TTL_DELTE_DAG", dag_module_str:"TTL"}, scheduled_task_count=0) [2024-09-26 16:01:10.636363] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:01:10.636363] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_MIGRATION", dag_net_count=0) [2024-09-26 16:01:10.636365] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_PREPARE_MIGRATION", dag_net_count=0) [2024-09-26 16:01:10.636368] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] dump_dag_status[DAG_NET](type="DAG_NET_COMPLETE_MIGRATION", dag_net_count=0) [2024-09-26 16:01:10.636370] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER", dag_net_count=0) [2024-09-26 16:01:10.636371] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_BACKUP", dag_net_count=0) [2024-09-26 16:01:10.636366] WDIAG [STORAGE] flush (ob_memtable.cpp:2064) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] memtable can not create dag successfully for long time(ls_id={id:1}, *this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, mt_stat_.ready_for_flush_time_=1727334209405104) [2024-09-26 16:01:10.636372] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_RESTORE", dag_net_count=0) [2024-09-26 16:01:10.636374] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] dump_dag_status[DAG_NET](type="DAG_NET_TYPE_BACKUP_CLEAN", dag_net_count=0) [2024-09-26 16:01:10.636373] WDIAG [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:651) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] memtable flush failed(tmp_ret=-4023, ls_->get_ls_id()={id:1}) [2024-09-26 16:01:10.636375] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1988) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status[DAG_NET](type="DAG_NET_TRANSFER_BACKFILL_TX", dag_net_count=0) [2024-09-26 16:01:10.636377] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1991) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=184) [2024-09-26 16:01:10.636378] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] add dag success(dag=0x7f9264c3ba20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1994541446894784994, dag_cnt=6, dag_type_cnts=5) [2024-09-26 16:01:10.636383] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3ba20) [2024-09-26 16:01:10.636385] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, create_flush_dag_time:1727337670636385, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209556884, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:226}, checkpoint_unit_ptr:0x7f92a1a73f00}, is_freeze_clock:true}) [2024-09-26 16:01:10.636391] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337670636375, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636393] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:10.636400] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337670636390) [2024-09-26 16:01:10.636404] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337670536176, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:10.636407] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3bf50, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8791616776033068452, dag_cnt=7, dag_type_cnts=6) [2024-09-26 16:01:10.636410] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c3bf50) [2024-09-26 16:01:10.636411] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337670636411, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405141, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847408}, checkpoint_unit_ptr:0x7f92a1a746d0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636415] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337670636398, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636425] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3c480, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6550230011160095889, dag_cnt=8, dag_type_cnts=7) [2024-09-26 16:01:10.636422] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223399872716, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.636427] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3c480) [2024-09-26 16:01:10.636429] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, create_flush_dag_time:1727337670636429, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405158, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1152921504606847409}, checkpoint_unit_ptr:0x7f92a1a75670}, is_freeze_clock:true}) [2024-09-26 16:01:10.636429] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:10.636442] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=13] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337670636422, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636448] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:10.636465] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:10.636467] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=21] add dag success(dag=0x7f9264c3c9b0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6809498982591245614, dag_cnt=9, dag_type_cnts=8) [2024-09-26 16:01:10.636468] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:10.636469] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3c9b0) [2024-09-26 16:01:10.636472] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, create_flush_dag_time:1727337670636472, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405174, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:200004}, checkpoint_unit_ptr:0x7f92a1a74ea0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636474] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.636476] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.636478] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337670636407) [2024-09-26 16:01:10.636475] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337670636463, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636486] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670636483, task_id:YB42C0A8002D-0006230056D8C758-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.636492] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=5] schedule one task(task=0x7f9264c3a3f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.636500] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337670636499, task_id:YB42C0A8002D-0006230056D8C759-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.636503] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=2] schedule one task(task=0x7f9264c3a920, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.636506] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337670636505, task_id:YB42C0A8002D-0006230056D8C75A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.636508] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=1] schedule one task(task=0x7f9264c3ae50, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.636511] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337670636511, task_id:YB42C0A8002D-0006230056D8C75B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.636513] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2] schedule one task(task=0x7f9264c3b380, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.636516] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670636515, task_id:YB42C0A8002D-0006230056D8C75C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.636518] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=2] schedule one task(task=0x7f9264c3b8b0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.636522] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337670636522, task_id:YB42C0A8002D-0006230056D8C75D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.636524] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=2] schedule one task(task=0x7f9264c3bde0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.636543] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c3cee0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6815944285528637599, dag_cnt=10, dag_type_cnts=9) [2024-09-26 16:01:10.636549] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] success to create and add dag(ret=0, dag=0x7f9264c3cee0) [2024-09-26 16:01:10.636547] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.636551] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, create_flush_dag_time:1727337670636551, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209506214, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:1}, checkpoint_unit_ptr:0x7f92a1a76610}, is_freeze_clock:true}) [2024-09-26 16:01:10.636555] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=8][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.636559] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:01:10.636558] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.636555] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337670636483, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636562] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.636564] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:01:10.636565] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3d410, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-4182959058374342994, dag_cnt=11, dag_type_cnts=10) [2024-09-26 16:01:10.636567] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3d410) [2024-09-26 16:01:10.636566] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.636564] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=9][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.636569] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.636569] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, create_flush_dag_time:1727337670636569, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405204, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:412}, checkpoint_unit_ptr:0x7f92a1a75e40}, is_freeze_clock:true}) [2024-09-26 16:01:10.636571] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.636574] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726163995122514000) [2024-09-26 16:01:10.636572] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847222}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:01:10.636572] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337670636562, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636581] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.636582] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3d940, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1586518267935340, dag_cnt=12, dag_type_cnts=11) [2024-09-26 16:01:10.636584] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3d940) [2024-09-26 16:01:10.636582] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=5] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.636585] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, create_flush_dag_time:1727337670636585, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405220, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:444}, checkpoint_unit_ptr:0x7f92a1a775b0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636588] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337670636579, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636597] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3de70, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3044367941103776476, dag_cnt=13, dag_type_cnts=12) [2024-09-26 16:01:10.636598] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264c3de70) [2024-09-26 16:01:10.636600] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, create_flush_dag_time:1727337670636600, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455964, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:309}, checkpoint_unit_ptr:0x7f92a1a76de0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636602] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.636604] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.636606] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.636610] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.636616] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.636603] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337670636595, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636620] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.636622] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726157244792838000) [2024-09-26 16:01:10.636624] INFO [STORAGE] ready_for_flush (ob_tx_data_memtable.cpp:865) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=1] memtable is frozen yet.(this=0x7f92a2294080) [2024-09-26 16:01:10.636626] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=18] add dag success(dag=0x7f9264c3e3a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1596830851104282047, dag_cnt=14, dag_type_cnts=13) [2024-09-26 16:01:10.636628] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3e3a0) [2024-09-26 16:01:10.636630] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, create_flush_dag_time:1727337670636630, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209455986, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:331}, checkpoint_unit_ptr:0x7f92a1a77d80}, is_freeze_clock:true}) [2024-09-26 16:01:10.636633] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337670636621, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636642] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3e8d0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2360195962081676673, dag_cnt=15, dag_type_cnts=14) [2024-09-26 16:01:10.636644] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3e8d0) [2024-09-26 16:01:10.636644] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.636646] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.636645] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, create_flush_dag_time:1727337670636645, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405262, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:458}, checkpoint_unit_ptr:0x7f92a1a7a490}, is_freeze_clock:true}) [2024-09-26 16:01:10.636648] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726164781026522000) [2024-09-26 16:01:10.636651] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.636648] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337670636640, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636654] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847223}, multi_version_start=1726164780922097000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726164780922097000) [2024-09-26 16:01:10.636657] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3ee00, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5675615614797020460, dag_cnt=16, dag_type_cnts=15) [2024-09-26 16:01:10.636659] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3ee00) [2024-09-26 16:01:10.636613] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861df20, is_inited:true, wash_score:1727337670636240000, hold_ref_cnt:true, ref_cnt:2, version:2, length:502, tablet_addr:[816017-1849-0-2305843642722988032-2](offset=1617920,size=590,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:226}, data_tablet_id:{id:226}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726178585269925002, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726178585269925002, cur_report_version:1726178585269925002, data_checksum:1151028828, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726201924319685000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861e5fc, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844296631750656-0](offset=1609728,size=1199,type=2,seq=0), type:"MAJOR", tablet_id:{id:226}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726178585269925002, v:0}}, snapshot_version:1726178585269925002, max_merge_version:1726178585269925002}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:5, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337670636375, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcc550, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241777443612000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209404959}, this:0x7f92a3bcc550, timestamp:1727334209404959, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:52, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:27262976}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277811954486000, v:0}, rec_scn:{val:1726241777645865000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843250807218176-2](offset=1613824,size=225,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176719032545-0](offset=1614049,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880960381-0](offset=1614205,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447480415-0](offset=1614431,size=193,type=2,seq=0)}}, extra_medium_info:{info:1, compat:1, last_compaction_type:0, wait_check_flag:0, last_medium_scn:1726178585269925002}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a73f00, 0x7f92a3bcc550, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:10.636661] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, create_flush_dag_time:1727337670636660, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405279, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:487}, checkpoint_unit_ptr:0x7f92a1a79cc0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636662] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.636663] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=50] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:226}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726178585269925002) [2024-09-26 16:01:10.636664] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337670636655, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636669] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241777443612000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.636672] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c3f330, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-612435342908088006, dag_cnt=17, dag_type_cnts=16) [2024-09-26 16:01:10.636674] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c3f330) [2024-09-26 16:01:10.636675] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, create_flush_dag_time:1727337670636675, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405296, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:455}, checkpoint_unit_ptr:0x7f92a1a794f0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636678] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337670636670, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636687] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3f860, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7868818205990873505, dag_cnt=18, dag_type_cnts=17) [2024-09-26 16:01:10.636688] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264c3f860) [2024-09-26 16:01:10.636690] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, create_flush_dag_time:1727337670636690, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405313, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:492}, checkpoint_unit_ptr:0x7f92a1a78550}, is_freeze_clock:true}) [2024-09-26 16:01:10.636693] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337670636685, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636702] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c3fd90, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7071731214611448811, dag_cnt=19, dag_type_cnts=18) [2024-09-26 16:01:10.636704] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264c3fd90) [2024-09-26 16:01:10.636705] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, create_flush_dag_time:1727337670636705, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405347, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:501}, checkpoint_unit_ptr:0x7f92a1a78d20}, is_freeze_clock:true}) [2024-09-26 16:01:10.636708] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337670636700, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636717] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c402c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8731750006000903454, dag_cnt=20, dag_type_cnts=19) [2024-09-26 16:01:10.636719] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c402c0) [2024-09-26 16:01:10.636721] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, create_flush_dag_time:1727337670636721, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405365, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:472}, checkpoint_unit_ptr:0x7f92a1a7ac60}, is_freeze_clock:true}) [2024-09-26 16:01:10.636724] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337670636715, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636733] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c407f0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6766388540859538791, dag_cnt=21, dag_type_cnts=20) [2024-09-26 16:01:10.636735] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c407f0) [2024-09-26 16:01:10.636736] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, create_flush_dag_time:1727337670636736, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405384, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:342}, checkpoint_unit_ptr:0x7f92a1a7cba0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636740] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337670636731, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636749] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c40d20, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=2986199386776543176, dag_cnt=22, dag_type_cnts=21) [2024-09-26 16:01:10.636751] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c40d20) [2024-09-26 16:01:10.636752] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337670636752, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405418, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:140}, checkpoint_unit_ptr:0x7f92a1a7b430}, is_freeze_clock:true}) [2024-09-26 16:01:10.636756] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337670636747, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636765] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c41250, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6947063304478012270, dag_cnt=23, dag_type_cnts=22) [2024-09-26 16:01:10.636770] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c41250) [2024-09-26 16:01:10.636772] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337670636772, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405436, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101009}, checkpoint_unit_ptr:0x7f92a1a7bc00}, is_freeze_clock:true}) [2024-09-26 16:01:10.636776] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337670636762, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636787] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c41780, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-5843378789243662202, dag_cnt=24, dag_type_cnts=23) [2024-09-26 16:01:10.636790] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c41780) [2024-09-26 16:01:10.636791] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, create_flush_dag_time:1727337670636791, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405453, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101010}, checkpoint_unit_ptr:0x7f92a1a7c3d0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636795] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337670636782, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636804] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c41cb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1841823684513746176, dag_cnt=25, dag_type_cnts=24) [2024-09-26 16:01:10.636806] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c41cb0) [2024-09-26 16:01:10.636808] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, create_flush_dag_time:1727337670636808, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405470, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:323}, checkpoint_unit_ptr:0x7f92a1a7d370}, is_freeze_clock:true}) [2024-09-26 16:01:10.636811] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337670636801, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636821] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c421e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-2678407261265885087, dag_cnt=26, dag_type_cnts=25) [2024-09-26 16:01:10.636823] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c421e0) [2024-09-26 16:01:10.636826] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, create_flush_dag_time:1727337670636826, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405487, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:377}, checkpoint_unit_ptr:0x7f92a1a7e310}, is_freeze_clock:true}) [2024-09-26 16:01:10.636829] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337670636818, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636839] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c42710, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7175089551615529855, dag_cnt=27, dag_type_cnts=26) [2024-09-26 16:01:10.636841] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c42710) [2024-09-26 16:01:10.636843] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, create_flush_dag_time:1727337670636842, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405503, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:297}, checkpoint_unit_ptr:0x7f92a1a7eae0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636846] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337670636836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636857] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=8] add dag success(dag=0x7f9264c42c40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3730526293242960203, dag_cnt=28, dag_type_cnts=27) [2024-09-26 16:01:10.636859] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c42c40) [2024-09-26 16:01:10.636861] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337670636861, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405519, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101065}, checkpoint_unit_ptr:0x7f92a1a7db40}, is_freeze_clock:true}) [2024-09-26 16:01:10.636864] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337670636854, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636873] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c43170, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=4663792628280492528, dag_cnt=29, dag_type_cnts=28) [2024-09-26 16:01:10.636875] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264c43170) [2024-09-26 16:01:10.636876] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337670636876, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405536, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:333}, checkpoint_unit_ptr:0x7f92a3bc4850}, is_freeze_clock:true}) [2024-09-26 16:01:10.636879] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337670636870, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636888] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c436a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8399557287176188414, dag_cnt=30, dag_type_cnts=29) [2024-09-26 16:01:10.636890] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264c436a0) [2024-09-26 16:01:10.636894] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, create_flush_dag_time:1727337670636894, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405552, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101066}, checkpoint_unit_ptr:0x7f92a3bc5020}, is_freeze_clock:true}) [2024-09-26 16:01:10.636898] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337670636885, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636908] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c43bd0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8068938909778324349, dag_cnt=31, dag_type_cnts=30) [2024-09-26 16:01:10.636910] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c43bd0) [2024-09-26 16:01:10.636912] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, create_flush_dag_time:1727337670636912, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405569, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:334}, checkpoint_unit_ptr:0x7f92a3bc57f0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636915] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337670636905, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636925] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c44100, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=7665372330263152188, dag_cnt=32, dag_type_cnts=31) [2024-09-26 16:01:10.636928] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c44100) [2024-09-26 16:01:10.636930] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, create_flush_dag_time:1727337670636929, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405586, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:451}, checkpoint_unit_ptr:0x7f92a3bc4080}, is_freeze_clock:true}) [2024-09-26 16:01:10.636933] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337670636922, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636942] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c44630, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6759548225246904317, dag_cnt=33, dag_type_cnts=32) [2024-09-26 16:01:10.636944] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264c44630) [2024-09-26 16:01:10.636945] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, create_flush_dag_time:1727337670636945, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405602, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:378}, checkpoint_unit_ptr:0x7f92a1a7f2b0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636949] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337670636939, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636959] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c44b60, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=8944614441376604790, dag_cnt=34, dag_type_cnts=33) [2024-09-26 16:01:10.636961] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c44b60) [2024-09-26 16:01:10.636963] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, create_flush_dag_time:1727337670636963, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209456003, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:121}, checkpoint_unit_ptr:0x7f92a3bc5fc0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636966] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337670636956, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.636975] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c45090, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=507863347806783454, dag_cnt=35, dag_type_cnts=34) [2024-09-26 16:01:10.636978] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c45090) [2024-09-26 16:01:10.636982] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, create_flush_dag_time:1727337670636982, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405633, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:215}, checkpoint_unit_ptr:0x7f92a3bc86d0}, is_freeze_clock:true}) [2024-09-26 16:01:10.636986] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337670636973, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.637010] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=9] add dag success(dag=0x7f9264c455c0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1604282355669434429, dag_cnt=36, dag_type_cnts=35) [2024-09-26 16:01:10.637013] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c455c0) [2024-09-26 16:01:10.637015] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, create_flush_dag_time:1727337670637015, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405649, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:456}, checkpoint_unit_ptr:0x7f92a3bc6790}, is_freeze_clock:true}) [2024-09-26 16:01:10.637019] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337670636995, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.637030] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c45af0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=3553649416163308074, dag_cnt=37, dag_type_cnts=36) [2024-09-26 16:01:10.637033] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] success to create and add dag(ret=0, dag=0x7f9264c45af0) [2024-09-26 16:01:10.637035] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337670637035, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405666, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:154}, checkpoint_unit_ptr:0x7f92a3bc6f60}, is_freeze_clock:true}) [2024-09-26 16:01:10.637038] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337670637026, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.637048] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c46020, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8097406237500329300, dag_cnt=38, dag_type_cnts=37) [2024-09-26 16:01:10.637050] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c46020) [2024-09-26 16:01:10.637054] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337670637054, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405682, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101015}, checkpoint_unit_ptr:0x7f92a3bc7730}, is_freeze_clock:true}) [2024-09-26 16:01:10.637058] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337670637045, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.637068] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c46550, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-3976350754940852513, dag_cnt=39, dag_type_cnts=38) [2024-09-26 16:01:10.637070] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] success to create and add dag(ret=0, dag=0x7f9264c46550) [2024-09-26 16:01:10.637071] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, create_flush_dag_time:1727337670637071, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405699, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101016}, checkpoint_unit_ptr:0x7f92a3bc7f00}, is_freeze_clock:true}) [2024-09-26 16:01:10.637074] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337670637065, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.637083] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c46a80, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-8720304653345806796, dag_cnt=40, dag_type_cnts=39) [2024-09-26 16:01:10.637085] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c46a80) [2024-09-26 16:01:10.637086] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, create_flush_dag_time:1727337670637086, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405716, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101063}, checkpoint_unit_ptr:0x7f92a3bca610}, is_freeze_clock:true}) [2024-09-26 16:01:10.637089] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337670637081, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.637099] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c46fb0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1830562711513853292, dag_cnt=41, dag_type_cnts=40) [2024-09-26 16:01:10.637101] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c46fb0) [2024-09-26 16:01:10.637103] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, create_flush_dag_time:1727337670637103, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405733, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:452}, checkpoint_unit_ptr:0x7f92a3bc9e40}, is_freeze_clock:true}) [2024-09-26 16:01:10.637107] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337670637096, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.637116] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c474e0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-1536604874905918360, dag_cnt=42, dag_type_cnts=41) [2024-09-26 16:01:10.637118] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c474e0) [2024-09-26 16:01:10.637120] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337670637120, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405750, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:328}, checkpoint_unit_ptr:0x7f92a3bc8ea0}, is_freeze_clock:true}) [2024-09-26 16:01:10.637123] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337670637113, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.637133] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c47a10, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-6933628521298738662, dag_cnt=43, dag_type_cnts=42) [2024-09-26 16:01:10.637135] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c47a10) [2024-09-26 16:01:10.637137] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337670637137, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405767, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:330}, checkpoint_unit_ptr:0x7f92a3bc9670}, is_freeze_clock:true}) [2024-09-26 16:01:10.637141] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337670637130, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.637150] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=6] add dag success(dag=0x7f9264c47f40, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=1056199536334423564, dag_cnt=44, dag_type_cnts=43) [2024-09-26 16:01:10.637153] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c47f40) [2024-09-26 16:01:10.637155] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, create_flush_dag_time:1727337670637155, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405784, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:329}, checkpoint_unit_ptr:0x7f92a3bcb5b0}, is_freeze_clock:true}) [2024-09-26 16:01:10.637158] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=3] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337670637147, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.637167] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c48470, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=-7405463607918801432, dag_cnt=45, dag_type_cnts=44) [2024-09-26 16:01:10.637170] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c48470) [2024-09-26 16:01:10.637174] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, create_flush_dag_time:1727337670637174, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405801, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:332}, checkpoint_unit_ptr:0x7f92a3bcade0}, is_freeze_clock:true}) [2024-09-26 16:01:10.637178] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337670637165, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.637188] INFO [COMMON] inner_add_dag (ob_dag_scheduler.cpp:3436) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] add dag success(dag=0x7f9264c489a0, start_time=0, id=Y0-0000000000000000-0-0, dag->hash()=6699723950958743903, dag_cnt=46, dag_type_cnts=45) [2024-09-26 16:01:10.637191] INFO [COMMON] create_and_add_dag (ob_dag_scheduler.h:1117) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=2] success to create and add dag(ret=0, dag=0x7f9264c489a0) [2024-09-26 16:01:10.637195] INFO [STORAGE.TRANS] operator() (ob_checkpoint_diagnose.cpp:265) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1] UpdateScheduleDagTime(info={ObCheckpointUnitDiagnoseInfo:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, create_flush_dag_time:1727337670637195, merge_finish_time:0, start_gc_time:0}, frozen_finish_time:1727334209405818, memtable_occupy_size:0, merge_start_time:0, occupy_size:608, concurrent_cnt:0, release_time:0}, param_={trace_id:1, ls_id:1, freeze_clock:1, key:{tablet_id:{id:101067}, checkpoint_unit_ptr:0x7f92a3bcbd80}, is_freeze_clock:true}) [2024-09-26 16:01:10.637199] INFO [STORAGE.TRANS] flush (ob_memtable.cpp:2057) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=4] schedule tablet merge dag successfully(ret=0, param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337670637185, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.637206] INFO [STORAGE] traversal_flush_ (ob_data_checkpoint.cpp:655) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=7] traversal_flush successfully(ls_->get_ls_id()={id:1}, flush_tasks=[{table:0x7f92a1a70850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a70080, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71020, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a717f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a71fc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72790, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a72f60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73730, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a73f00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a746d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75670, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a74ea0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76610, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a75e40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a775b0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a76de0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a77d80, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7a490, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a79cc0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a794f0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78550, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a78d20, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7ac60, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7cba0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7b430, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7bc00, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7c3d0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7d370, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7e310, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7eae0, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a1a7db40, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}, {table:0x7f92a3bc4850, t3m_:0x7f92cfbda030, allocator_:null, table_type_:0, meta_handle:{phy_addr:[0-0-0-4611686018427387904-0](offset=0,size=0,type=4,seq=0), io_handle:{macro_handle:{macro_id:[9223372036854775807](ver=0,mode=0,seq=0), io_handle:{io_request:"NULL"}}}, cache_handle:{cache_value:NULL, allocator:null}}, lifetime_guaranteed_by_tablet:false}]) [2024-09-26 16:01:10.637251] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [2934][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=43] succeed to traversal_flush(ret=0, ls_cnt=1) [2024-09-26 16:01:10.644509] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.644521] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200489}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:01:10.644526] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=16] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337670636278, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:01:10.644532] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=6] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.644534] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.644631] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=13] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:01:10.644637] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.644648] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847222}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:01:10.644675] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=18] succeed to build merge ctx(tablet_id={id:1152921504606847222}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847222}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670636534, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=8.11ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.644937] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=43] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1857, total_rows=90993) [2024-09-26 16:01:10.645612] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=5] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=8366, total_rows=58562) [2024-09-26 16:01:10.645985] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337670636375, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.645986] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.646000] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=15] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752755184, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828752755184, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.646011] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=25] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337670636294, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=2, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:01:10.646020] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=14] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.646024] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.646020] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.646031] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:226}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752755184, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.646053] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:226}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752755184, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752755184, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:226}, table_count:1, [{i:0, table_key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}, merge_scn:{val:1726241777443612000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670636599, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=9.43ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.646075] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=6] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75B-0-0, dag_ret:0, dag_status:2, start_time:1727337670636513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505470, output_block_cnt_arr:0x7f9266505478, concurrent_cnt:1, estimate_row_cnt:329211, estimate_occupy_size:150994944, latest_update_ts:1727337670636534, estimated_finish_time:1727337687646146}) [2024-09-26 16:01:10.646084] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=24] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75D-0-0, dag_ret:0, dag_status:2, start_time:1727337670636524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f480, output_block_cnt_arr:0x7f9264b1f488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:56623104, latest_update_ts:1727337670636599, estimated_finish_time:1727337687646110}) [2024-09-26 16:01:10.646088] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f9264c3b380, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75B-0-0, dag_ret:0, dag_status:2, start_time:1727337670636513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.646091] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3bde0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75D-0-0, dag_ret:0, dag_status:2, start_time:1727337670636524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:01:10.646096] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=7] task finish process(ret=0, start_time=1727337670636530, end_time=1727337670646095, runtime=9565, *this={this:0x7f9264c3b380, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75B-0-0, dag_ret:0, dag_status:2, start_time:1727337670636513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.646097] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=5] task finish process(ret=0, start_time=1727337670636597, end_time=1727337670646097, runtime=9500, *this={this:0x7f9264c3bde0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75D-0-0, dag_ret:0, dag_status:2, start_time:1727337670636524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:01:10.646129] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.646137] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=5] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.646205] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:01:10.646211] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.646200] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=226, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665fa060, count:5, allocator:0x7f92691aa7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:01:10.646218] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847223}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:01:10.646239] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:1152921504606847223}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f92685e7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847223}, table_count:2, [{i:0, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670636641, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=9.57ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.646595] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=22] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1852, total_rows=90748) [2024-09-26 16:01:10.647247] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=5] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=7999, total_rows=55993) [2024-09-26 16:01:10.647431] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684e2060, count:4, allocator:0x7f92636107d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684e2350, local_datums:0x7f92684e2350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:10.647454] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=20] multi version scan iterator init succ(param.table_id_=1152921504606847222, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268564060, count:4, allocator:0x7f9263611558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9268564350, local_datums:0x7f9268564350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:10.647710] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=26][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:10.647770] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.647775] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.647785] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=10][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:10.647787] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:10.647790] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f926856c140, hex: 3230323430393131303133333438, cstr: 20240911013348},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f926856c14e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f926856c0e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f926856c118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3080, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7221, data_checksum:4284205003, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9268596030, buf_size:7221, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1004563202, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:01:10.647805] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=15][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:10.647807] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:10.647809] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:10.647811] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:10.647813] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684e2350, local_datums:0x7f92684e2350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e23d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684e2350, local_datums:0x7f92684e2350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e23d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9263610118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263610118, limit_param:null, stmt_allocator:0x7f92636107d8, allocator:0x7f92636107d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684e2350, local_datums:0x7f92684e2350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e23d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:01:10.647872] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=59][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:10.647874] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847222}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684e2350, local_datums:0x7f92684e2350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e23d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f9263610118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263610118, limit_param:null, stmt_allocator:0x7f92636107d8, allocator:0x7f92636107d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684e2350, local_datums:0x7f92684e2350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f2c5f4c, hex: 3230323430393131303133333438, cstr: 20240911013348},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f2c5f5a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e23d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684e2408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013348", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f2c5e48 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f2c5ed8 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:01:10.647905] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=31][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:10.647908] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72790, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402823}, this:0x7f92a1a72790, timestamp:1727334201402823, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:7, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405070, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.647922] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=10] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337670636278, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.647949] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=22][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:10.647962] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:10.647966] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:10.647968] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337670647379, end_time=1727337670647968, runtime=589, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75B-0-0, dag_ret:0, dag_status:2, start_time:1727337670636513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.647976] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75B-0-0, dag_ret:0, dag_status:2, start_time:1727337670636513, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:10.647985] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=11471, dag_cnt=45, dag_cnts_[dag.get_type()]=44, &dag=0x7f9264c3afc0, dag={ObIDag:{this:0x7f9264c3afc0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670636513, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1288986994123435108}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847222}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.647996] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75B-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670636511, task_id:YB42C0A8002D-0006230056D8C75B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847222;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.648064] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75C-0-0, dag_ret:0, dag_status:2, start_time:1727337670636518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92685e7470, output_block_cnt_arr:0x7f92685e7478, concurrent_cnt:1, estimate_row_cnt:332829, estimate_occupy_size:150994944, latest_update_ts:1727337670636641, estimated_finish_time:1727337687648135}) [2024-09-26 16:01:10.648076] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=11] succeed to init merge ctx(task={this:0x7f9264c3b8b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75C-0-0, dag_ret:0, dag_status:2, start_time:1727337670636518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.648082] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=5] task finish process(ret=0, start_time=1727337670636639, end_time=1727337670648082, runtime=11443, *this={this:0x7f9264c3b8b0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75C-0-0, dag_ret:0, dag_status:2, start_time:1727337670636518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.648123] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200240}, multi_version_start=1726159999077775000, min_reserved_snapshot=0, min_medium_snapshot=1726159999077775000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726158220122116000) [2024-09-26 16:01:10.648139] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=8] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.648767] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1, storage_schema:{this:0x7f9264b212b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.648784] INFO [STORAGE] prepare_tx_data_list (ob_tx_data_memtable.cpp:266) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=16] construct list more than once. skip this time(this=0x7f92a2294080) [2024-09-26 16:01:10.652817] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=18][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:10.652888] ERROR issue_dba_error (ob_log.cpp:1875) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=11][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.652893] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.652903] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=10][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:10.652906] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:10.652910] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=3][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e80a8, hex: 0000000000000000, int: 0},idx=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92685e80e0},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8118, hex: 58A3C56A4A280BE8, int: -1726241732082425000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685e8150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-19590, column_count:10, rowkey_column_count:5, has_column_checksum:0, row_count:224, row_store_type:0, opt:0, var_column_count:0, row_offset:15527, original_length:16363, max_merged_trans_version:1726241777443612000, data_length:16363, data_zlength:16363, data_checksum:2057046089, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b7004090, buf_size:16363, data_size:16363, row_count:224, column_count:10, max_merged_trans_version:1726241777443612000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1006468742, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16363}) [2024-09-26 16:01:10.652926] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=15][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:10.652928] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=1][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:10.652929] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:10.652930] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:01:10.652932] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:10.652934] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:10.652935] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fa3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fa3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f92691aa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691aa118, limit_param:null, stmt_allocator:0x7f92691aa7d8, allocator:0x7f92691aa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fa3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337670636375, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}) [2024-09-26 16:01:10.653010] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=75][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:10.653012] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:226}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1710828752755184, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fa3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, store_ctx:{this:0x7f92691aa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691aa118, limit_param:null, stmt_allocator:0x7f92691aa7d8, allocator:0x7f92691aa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241777443612000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241777443612000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92665fa360, local_datums:0x7f92665fa360}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa370, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa3a8, hex: 0000000000000000, int: 0},col_id=2:{len: 0, flag: 0, null: 0, ptr: 0x7f92665fa3e0},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa418, hex: 58A3C56A4A280BE8, int: -1726241732082425000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fa450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d6e, hex: F9BA3CF901220600, int: 1726241732082425},col_id=7:NOP,col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d76, hex: 78083E89B5D7F417, int: 1726241731881339000},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f926a4e0d7e, hex: 78083E89B5D7F417, int: 1726241731881339000},]}, key:{"BIGINT":1},{"BIGINT":0},{"VARCHAR":"", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927dc040c8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=1215 list_head=0x7f926a4cafd8 latest_compact_node=0x7f926a28e478 max_trans_version={val:1726241777443612000, v:0} max_trans_id=125584734 max_elr_trans_version={val:1726241777443612000, v:0} max_elr_trans_id=125584734 latest_compact_ts=1727334209409175 last_compact_cnt=142659 total_trans_node_cnt=306907 max_modify_scn={val:1726241777443612000, v:0} min_modify_scn={val:1726178585792510000, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f926a4c81e8 trans_version={val:1726241731881339000, v:0} scn={val:1726241731881339000, v:0} tx_id={txid:125584382} prev=0x7f926a04a890 next=0x7f926a4e0cf8 modify_count=145602 acc_checksum=1662313619 version=1726211867506969 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:57} seq_no=1726241731747450, max_committed_trans_version:1726241777443612000, cur_trans_version:{val:1726241732082425000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241777443612000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:224, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337670636375, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:10, cur_pos:0, count:0}}]) [2024-09-26 16:01:10.653051] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=39][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:10.653055] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=2] partition merge iter row count(i=0, row_count=224, ghost_row_count=0, pkey={tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, table={ObITable:{this:0x7f92a1a73f00, key:{tablet_id:{id:226}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201764874}, this:0x7f92a1a73f00, timestamp:1727334201764874, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752755184, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:9, clock:419430400}, host:0x7f92ba380030, arena_handle:{allocated:56623104}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241777443612000, v:0}, rec_scn:{val:1726178585792510000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209404953, mt_stat_.ready_for_flush_time:1727334209556869, mt_stat_.create_flush_dag_time:1727337670636375, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.653074] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=11][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:10.653088] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=3][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:10.653092] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:10.653094] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337670646147, end_time=1727337670653094, runtime=6947, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75D-0-0, dag_ret:0, dag_status:2, start_time:1727337670636524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}) [2024-09-26 16:01:10.653102] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75D-0-0, dag_ret:0, dag_status:2, start_time:1727337670636524, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:10.653112] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=6] dag finished(dag_ret=-4184, runtime=16588, dag_cnt=44, dag_cnts_[dag.get_type()]=43, &dag=0x7f9264c3ba20, dag={ObIDag:{this:0x7f9264c3ba20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670636524, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1994541446894784994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:226}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241777443612000, v:0}}}}) [2024-09-26 16:01:10.653123] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75D-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337670636522, task_id:YB42C0A8002D-0006230056D8C75D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=226;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.653261] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.653275] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=14] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337670636259, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=17, max_schema_version_on_memtable=1725120002010696) [2024-09-26 16:01:10.653283] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=17, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.653379] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1887, total_rows=92463) [2024-09-26 16:01:10.653386] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=6] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.653395] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200489}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:19, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.653419] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:200489}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:17, store_column_cnt:17, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200489}, table_count:2, [{i:0, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670636552, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=16.84ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.653725] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=24] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75A-0-0, dag_ret:0, dag_status:2, start_time:1727337670636507, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab460, output_block_cnt_arr:0x7f92684ab468, concurrent_cnt:1, estimate_row_cnt:254187, estimate_occupy_size:138412032, latest_update_ts:1727337670636552, estimated_finish_time:1727337687653790}) [2024-09-26 16:01:10.653740] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=14] succeed to init merge ctx(task={this:0x7f9264c3ae50, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75A-0-0, dag_ret:0, dag_status:2, start_time:1727337670636507, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.653746] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=5] task finish process(ret=0, start_time=1727337670636550, end_time=1727337670653746, runtime=17196, *this={this:0x7f9264c3ae50, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75A-0-0, dag_ret:0, dag_status:2, start_time:1727337670636507, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.653767] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=2] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.653773] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=5] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.653781] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670653779, task_id:YB42C0A8002D-0006230056D8C75E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.653786] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=5] schedule one task(task=0x7f9264c3c310, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.653791] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670653790, task_id:YB42C0A8002D-0006230056D8C75F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.653793] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=2] schedule one task(task=0x7f9264c3c840, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.653894] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b0060, count:3, allocator:0x7f92b32387d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92684b0320, local_datums:0x7f92684b0320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:01:10.653916] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=19] multi version scan iterator init succ(param.table_id_=200489, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:17, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:19}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:3, allocator:0x7f92b3239558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,]}) [2024-09-26 16:01:10.653952] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.653958] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.653962] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.653970] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.654062] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d32060, count:4, allocator:0x7f92691aa7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:10.654082] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=17] multi version scan iterator init succ(param.table_id_=1152921504606847223, range=0x7f92aadd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:4, allocator:0x7f92691ab558, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9264a62350, local_datums:0x7f9264a62350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:10.654102] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=9][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:10.654166] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.654170] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.654178] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:10.654181] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:10.654183] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665a6070, hex: 9F094D0300000000, int: 55380383},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665a60a8, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665a60e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:3044, column_count:19, rowkey_column_count:3, has_column_checksum:0, row_count:83, row_store_type:0, opt:5, var_column_count:0, row_offset:16109, original_length:16381, max_merged_trans_version:1726163995623997000, data_length:16381, data_zlength:7759, data_checksum:3416849541, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9268428030, buf_size:7759, data_size:16381, row_count:83, column_count:19, max_merged_trans_version:1726163995623997000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3537734793, row_count_delta:83, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16381}) [2024-09-26 16:01:10.654197] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=14][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:10.654202] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=5][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:10.654204] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:10.654206] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:10.654207] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b44f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b45d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b46e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b44f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b45d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b46e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f92b3238e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b3238e98, limit_param:null, stmt_allocator:0x7f92b3239558, allocator:0x7f92b3239558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b44f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b45d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b46e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337670636259, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}) [2024-09-26 16:01:10.654275] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=68][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:10.654277] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200489}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b44f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b45d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b46e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, store_ctx:{this:0x7f92b3238e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92b3238e98, limit_param:null, stmt_allocator:0x7f92b3239558, allocator:0x7f92b3239558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726163995623997000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:19, datum_buffer:{capacity:32, datums:0x7f92684b4320, local_datums:0x7f92684b4320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4330, hex: 9F094D0300000000, int: 55380383},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4368, hex: B841C3D8FD6E0BE8, int: -1726163995623997000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a0, hex: 0000000000000000, int: 0},col_id=3:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c950, hex: 343132333330373432, cstr: 412330742},col_id=4:{len: 9, flag: 0, null: 0, ptr: 0x7f929b86c959, hex: 343132333330373432, cstr: 412330742},col_id=5:{len: 14, flag: 0, null: 0, ptr: 0x7f929b86c962, hex: 3230323430393133303135393137, cstr: 20240913015917},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c970, hex: A54E4013611B5E40, int: 4638174769986948773},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c978, hex: 4A7B832F4CE24240, int: 4630512183732829002},col_id=8:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b44f0},col_id=9:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4528, hex: 0000000000000000, int: 0},col_id=10:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4560, hex: 0000000000000000, int: 0},col_id=11:null,col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b45d0, hex: 2101000000000000, int: 289},col_id=13:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c984, hex: 3333333333331F40, int: 4620468037700760371},col_id=14:null,col_id=15:null,col_id=16:{len: 24, flag: 0, null: 0, ptr: 0x7f929b86c98c, hex: 70726F766973696F6E616C5F4E4445794D7A4D774E7A5179, cstr: provisional_NDEyMzMwNzQy},col_id=17:{len: 0, flag: 0, null: 0, ptr: 0x7f92684b46e8},col_id=18:{len: 8, flag: 0, null: 0, ptr: 0x7f929b86c9a4, hex: C0FCBEDFEF210600, int: 1726163995000000},]}, key:{"BIGINT UNSIGNED":55380383}, value_iter:{value:{this=0x7f929b86c848 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f929b86c8d8 latest_compact_node=(nil) max_trans_version={val:1726163995623997000, v:0} max_trans_id=124981180 max_elr_trans_version={val:1726163995623997000, v:0} max_elr_trans_id=124981180 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726163995614630001, v:0} min_modify_scn={val:1726163995614630001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726163995623997000, cur_trans_version:{val:1726163995623997000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:83, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337670636259, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:19, cur_pos:0, count:0}}]) [2024-09-26 16:01:10.654318] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=41][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:10.654322] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=3] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a73730, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170043069079001, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201404273}, this:0x7f92a1a73730, timestamp:1727334201404273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:5, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:31457280}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170055810024002, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405104, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.654335] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=10] partition merge iter row count(i=1, row_count=83, ghost_row_count=0, pkey={tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, table={ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337670636259, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.654344] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:10.654352] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=8][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:10.654355] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:10.654359] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:10.654362] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337670653806, end_time=1727337670654361, runtime=555, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75A-0-0, dag_ret:0, dag_status:2, start_time:1727337670636507, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.654369] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75A-0-0, dag_ret:0, dag_status:2, start_time:1727337670636507, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:10.654377] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=17869, dag_cnt=43, dag_cnts_[dag.get_type()]=42, &dag=0x7f9264c3aa90, dag={ObIDag:{this:0x7f9264c3aa90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670636507, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-9069613569758001279}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200489}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.654387] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75A-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337670636505, task_id:YB42C0A8002D-0006230056D8C75A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200489;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.654396] ERROR issue_dba_error (ob_log.cpp:1875) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.654400] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.654407] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:10.654409] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:10.654411] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 14, flag: 0, null: 0, ptr: 0x7f9287dd2140, hex: 3230323430393131303133333131, cstr: 20240911013311},idx=1:{len: 9, flag: 0, null: 0, ptr: 0x7f9287dd214e, hex: 373036303030303035, cstr: 706000005},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd20e0, hex: 20C459BD3B620BE8, int: -1726178023447084000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287dd2118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-15038, column_count:4, rowkey_column_count:4, has_column_checksum:0, row_count:269, row_store_type:0, opt:4, var_column_count:0, row_offset:15349, original_length:16365, max_merged_trans_version:1726190651576080000, data_length:16365, data_zlength:7227, data_checksum:1452123993, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9265a74030, buf_size:7227, data_size:16365, row_count:269, column_count:4, max_merged_trans_version:1726190651576080000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1818264584, row_count_delta:-269, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16365}) [2024-09-26 16:01:10.654424] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=13][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:10.654426] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:10.654427] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:10.654429] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:10.654428] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.654433] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.654435] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.654442] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=3] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.654434] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=5][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32408, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92691aa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691aa118, limit_param:null, stmt_allocator:0x7f92691aa7d8, allocator:0x7f92691aa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}) [2024-09-26 16:01:10.654516] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=82][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:10.654522] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670654520, task_id:YB42C0A8002D-0006230056D8C760-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.654527] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=4] schedule one task(task=0x7f9264c3cd70, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.654545] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.654518] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847223}, iter_end:false, schema_rowkey_column_cnt:2, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32408, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92691aa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691aa118, limit_param:null, stmt_allocator:0x7f92691aa7d8, allocator:0x7f92691aa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"DELETE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726178023447084000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:{len: 14, flag: 0, null: 0, ptr: 0x7f927f0a4efc, hex: 3230323430393131303133333131, cstr: 20240911013311},col_id=1:{len: 9, flag: 0, null: 0, ptr: 0x7f927f0a4f0a, hex: 373036303030303035, cstr: 706000005},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323d0, hex: 20C459BD3B620BE8, int: -1726178023447084000},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32408, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"20240911013311", collation:"utf8mb4_general_ci"},{"VARCHAR":"706000005", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f927f0a4df8 latch_=unlocked flag=3 first_dml=DELETE last_dml=DELETE update_since_compact=1 list_head=0x7f927f0a4e88 latest_compact_node=(nil) max_trans_version={val:1726178023447084000, v:0} max_trans_id=125090832 max_elr_trans_version={val:1726178023447084000, v:0} max_elr_trans_id=125090832 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726178023079620001, v:0} min_modify_scn={val:1726178023079620001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726178023447084000, cur_trans_version:{val:1726178023447084000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:184, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:4, cur_pos:0, count:0}}]) [2024-09-26 16:01:10.654549] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.654550] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=32][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:10.654551] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.654554] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.654552] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=1] partition merge iter row count(i=0, row_count=184, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a72f60, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201403026}, this:0x7f92a1a72f60, timestamp:1727334201403026, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:8, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:44040192}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405015, mt_stat_.ready_for_flush_time:1727334209405087, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.654564] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=9] partition merge iter row count(i=1, row_count=86, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a717f0, key:{tablet_id:{id:1152921504606847223}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199757827}, this:0x7f92a1a717f0, timestamp:1727334199757827, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:4, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401496, mt_stat_.ready_for_flush_time:1727334201452922, mt_stat_.create_flush_dag_time:1727337670636294, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.654575] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:10.654587] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:10.654590] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:10.654592] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337670654010, end_time=1727337670654592, runtime=582, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75C-0-0, dag_ret:0, dag_status:2, start_time:1727337670636518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.654558] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8621da0, is_inited:true, wash_score:1727337670636240000, hold_ref_cnt:true, ref_cnt:2, version:2, length:480, tablet_addr:[816017-1849-0-2305843619101044736-1](offset=1994752,size=568,type=2,seq=1), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:200004}, data_tablet_id:{id:200004}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1082501607, row_count:1}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:1710828761605424, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:1710828761605424}, table_store_addr:{ptr:{this:0x7f92b8622434, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844140939550720-0](offset=1974272,size=1054,type=2,seq=0), type:"MAJOR", tablet_id:{id:200004}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077597769620000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843906863834142-0](offset=1975326,size=836,type=2,seq=0), type:"MINOR", tablet_id:{id:200004}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178584585241000, upper_trans_version:1726178584585241000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337670636463, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697961984-1](offset=1982464,size=332,type=2,seq=1)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843163834499404-0](offset=1982796,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843252955070940-0](offset=1982940,size=227,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828761613515000, v:0}, create_commit_version:1710828761613515000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a74ea0, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:01:10.654599] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75C-0-0, dag_ret:0, dag_status:2, start_time:1727337670636518, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:10.654601] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=43] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:200004}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.654612] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.654609] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=7] dag finished(dag_ret=-4184, runtime=18091, dag_cnt=42, dag_cnts_[dag.get_type()]=41, &dag=0x7f9264c3b4f0, dag={ObIDag:{this:0x7f9264c3b4f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670636518, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5355160120942094702}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847223}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726164780922097000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.654617] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75C-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670636515, task_id:YB42C0A8002D-0006230056D8C75C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847223;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.654647] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337670654645, task_id:YB42C0A8002D-0006230056D8C761-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.654651] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=4] schedule one task(task=0x7f9264c3d2a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.654649] INFO [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:329) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=3] pre-process commit versions row(get_ls_id()={id:1}, do_recycle_=true, current_time=1727337670654647, prev_recycle_time=0) [2024-09-26 16:01:10.654669] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.654672] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.654673] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.654680] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=5] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.654683] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.654688] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241621541056000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.657326] INFO [STORAGE] get_ls_min_end_scn (ob_ls_tablet_service.cpp:5618) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=7] get ls min end scn finish(ls_checkpoint={val:1726163586617842000, v:0}) [2024-09-26 16:01:10.657342] WDIAG [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:761) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=12][errcode=-4023] failed to get snapshot version(ret=-4023, MTL_ID()=1) [2024-09-26 16:01:10.657347] INFO [STORAGE] get_recycle_scn (ob_tx_data_table.cpp:773) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=4] get tx data recycle scn finish.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, recycle_scn={val:1726076389100953000, v:0}, min_end_scn_from_old_tablets={val:4611686018427387903, v:0}, min_end_scn_from_latest_tablets={val:1726076389100953000, v:0}) [2024-09-26 16:01:10.657352] WDIAG [STORAGE] pre_process_commit_version_row_ (ob_tx_data_memtable.cpp:336) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=4][errcode=-4023] get recycle ts failed.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:01:10.657359] WDIAG [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:236) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=5][errcode=-4023] process commit version row failed.(ret=-4023, ret="OB_EAGAIN", this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337670636144, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:01:10.657373] INFO [STORAGE] pre_process_for_merge (ob_tx_data_memtable.cpp:254) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=13] [TX DATA MERGE]pre process for merge done(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}, this=0x7f92a2294080, tg=time guard have no click events for optimization, fake_tx_data_guard={tx_data:{tx_id:{txid:9223372036854775807}, ref_cnt:1, state:"RUNNING", commit_version:{val:18446744073709551615, v:3}, start_scn:{val:18446744073709551615, v:3}, end_scn:{val:18446744073709551615, v:3}, undo_status_list:{head:null, undo_node_cnt:0{}}}}, this={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337670636144, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:01:10.657385] WDIAG [STORAGE.COMPACTION] pre_process_tx_data_table_merge_ (ob_tx_table_merge_task.cpp:117) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=11][errcode=-4023] do pre process for tx data table merge failed.(ret=-4023, ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, table={ObITable:{this:0x7f92a2294080, key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref_cnt:5, upper_trans_version:-4007, timestamp:0}, is_inited:true, is_iterating:false, pre_process_done:false, construct_list_done:true, min_tx_scn:{val:1726163586617842000, v:0}, max_tx_scn:{val:1726171772514786000, v:0}, min_start_scn:{val:1726163586617842000, v:0}, snapshot_version:{val:1726163586617842000, v:0}, inserted_cnt:53731, deleted_cnt:0, write_ref:0, occupied_size:0, state:2, stat_change_ts:{frozen_time:1727334201401260, ready_for_flush_time:1727334204304509, create_flush_dag_time:1727337670636144, release_time:0}, tx_data_map:0x7f92a6806060, memtable_mgr:0x7f92a6c041b0, commit_versions_serialize_size:0, row_key_array:[], tx_id_2_range:[]}) [2024-09-26 16:01:10.657393] WDIAG [STORAGE.COMPACTION] inner_init_ctx (ob_tx_table_merge_task.cpp:157) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=8][errcode=-4023] pre process tx data table for merge failed.(ret=-4023, ret="OB_EAGAIN", ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.657398] WDIAG [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1061) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=3][errcode=-4023] fail to inner init ctx(ret=-4023, tablet_id={id:49402}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, create_snapshot_version:0, is_full_merge:false, merge_level:1, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:4, range_array:[], concurrent_cnt:0, is_inited:false}, schema_ctx:{schema_version:1, storage_schema:{this:0x7f9264b212b0, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:0, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:49402}, table_count:1, [{i:0, table_key:{tablet_id:{id:49402}, column_group_idx:0, table_type:"TX_DATA_MEMTABLE", scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}, ref:4}]}, schedule_major:false, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}, merge_scn:{val:4611686018427387903, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:NULL, compaction_filter:NULL, time_guard:total=20.78ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}) [2024-09-26 16:01:10.657433] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:975) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=34][errcode=-4023] failed to build merge ctx(ret=-4023, ctx->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.657439] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:994) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=6][errcode=-4023] sstable merge finish(ret=-4023, ctx=0x7f9264b20060, task={this:0x7f9264c3a3f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C758-0-0, dag_ret:0, dag_status:2, start_time:1727337670636491, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:01:10.657447] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=8][errcode=-4023] failed to process task(ret=-4023) [2024-09-26 16:01:10.657449] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=2] task finish process(ret=-4023, start_time=1727337670636605, end_time=1727337670657449, runtime=20844, *this={this:0x7f9264c3a3f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C758-0-0, dag_ret:0, dag_status:2, start_time:1727337670636491, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}) [2024-09-26 16:01:10.657455] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=6][errcode=-4023] failed to do work(ret=-4023, *task_={this:0x7f9264c3a3f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C758-0-0, dag_ret:0, dag_status:2, start_time:1727337670636491, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:10.657463] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=5] dag finished(dag_ret=-4023, runtime=20971, dag_cnt=41, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264c3a030, dag={ObIDag:{this:0x7f9264c3a030, type:3, name:"TX_TABLE_MERGE", id:YB42C0A8002D-0006230056D8C758-0-0, dag_ret:-4023, dag_status:5, start_time:1727337670636491, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6163825430799493735}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:49402}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:9223372036854775807, base_version:0, snapshot_version:1726163586617842000}, scn_range:{start_scn:{val:1726163586417012001, v:0}, end_scn:{val:1726171772514786000, v:0}}}}) [2024-09-26 16:01:10.657472] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2865][T1_TX_TABLE_MER][T1][YB42C0A8002D-0006230056D8C758-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670636483, task_id:YB42C0A8002D-0006230056D8C758-0-0, task_type:4, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_TX_TABLE_MERGE";ls_id=1;tablet_id=49402;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.657487] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670657484, task_id:YB42C0A8002D-0006230056D8C762-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.657505] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=17] schedule one task(task=0x7f9264c3d7d0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.657529] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.657533] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.657538] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.657543] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.657547] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726176799210367000, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b862fe60, is_inited:true, wash_score:1727337670636240000, hold_ref_cnt:true, ref_cnt:2, version:2, length:497, tablet_addr:[816017-1849-0-2305843637353291776-2](offset=630784,size=585,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:412}, data_tablet_id:{id:412}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726176799210367000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3559841642, row_count:4}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8630518, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844255828574208-0](offset=622592,size=1161,type=2,seq=0), type:"MAJOR", tablet_id:{id:412}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725843390668168000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335631832201-0](offset=623753,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:412}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337670636562, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd6790, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334211402085}, this:0x7f92a3bd6790, timestamp:1727334211402085, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:70, clock:618659840}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277807335692000, v:0}, rec_scn:{val:1726250403931267000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365696606208-2](offset=626688,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843170275594572-0](offset=627020,size=150,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251879973346-0](offset=627170,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216446493380-0](offset=627396,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a75e40, 0x7f92a3bd6790, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:10.657594] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=47] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:412}, multi_version_start=1726176799210367000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.657602] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.657689] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.657702] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=13] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337670636210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=18, max_schema_version_on_memtable=1711680543204904) [2024-09-26 16:01:10.657710] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1711680543204904, storage_schema:{this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1711680543204904, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.657938] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=9] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:01:10.657946] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=7] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.657955] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200240}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1711680543204904, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.657980] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:200240}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1711680543204904, storage_schema:{this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1711680543204904, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:6, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:2, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200240}, table_count:2, [{i:0, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref:3}{i:1, table_key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8637b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670636558, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=21.39ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.658314] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=58] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=5143, total_rows=36001) [2024-09-26 16:01:10.658886] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C759-0-0, dag_ret:0, dag_status:2, start_time:1727337670636502, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263b1f460, output_block_cnt_arr:0x7f9263b1f468, concurrent_cnt:1, estimate_row_cnt:85417, estimate_occupy_size:134217728, latest_update_ts:1727337670636558, estimated_finish_time:1727337687658949}) [2024-09-26 16:01:10.658900] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f9264c3a920, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C759-0-0, dag_ret:0, dag_status:2, start_time:1727337670636502, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.658938] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=37] task finish process(ret=0, start_time=1727337670636547, end_time=1727337670658937, runtime=22390, *this={this:0x7f9264c3a920, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C759-0-0, dag_ret:0, dag_status:2, start_time:1727337670636502, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.658957] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=4] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.659045] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92a98d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:3, allocator:0x7f92bc9127d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:01:10.659067] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=20] multi version scan iterator init succ(param.table_id_=200240, range=0x7f92a98d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d32060, count:3, allocator:0x7f92bc913558, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287d32320, local_datums:0x7f9287d32320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:01:10.659280] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=9][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:10.659330] ERROR issue_dba_error (ob_log.cpp:1875) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.659334] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf2041c8 0xf217c49 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.659344] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=9][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:10.659346] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:10.659351] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=5][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9265b04108, hex: 313030303030333830, cstr: 100000380},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b040a8, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9265b040e0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-29413, column_count:20, rowkey_column_count:3, has_column_checksum:0, row_count:100, row_store_type:0, opt:0, var_column_count:0, row_offset:16082, original_length:16422, max_merged_trans_version:1726189672100745000, data_length:16454, data_zlength:5272, data_checksum:4171874375, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f9287dce030, buf_size:5272, data_size:16454, row_count:100, column_count:20, max_merged_trans_version:1726189672100745000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:371002229, row_count_delta:22, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16422}) [2024-09-26 16:01:10.659365] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=14][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:10.659369] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=4][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:10.659372] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:10.659373] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:10.659375] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1526) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, partition_fuser_->get_result_row()={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a60148, local_datums:0x7f9264a60148}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a60190, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a601c8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a603c0, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a603f8, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, fuse_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a625d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92bc912118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92bc912118, limit_param:null, stmt_allocator:0x7f92bc9127d8, allocator:0x7f92bc9127d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a625d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287d32320, local_datums:0x7f9287d32320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d325d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92bc912e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92bc912e98, limit_param:null, stmt_allocator:0x7f92bc913558, allocator:0x7f92bc913558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287d32320, local_datums:0x7f9287d32320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d325d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337670636210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:01:10.659501] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=126][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a625d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92bc912118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92bc912118, limit_param:null, stmt_allocator:0x7f92bc9127d8, allocator:0x7f92bc9127d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726180870236146000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a62320, local_datums:0x7f9264a62320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f92793bf184, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62368, hex: B0F67FEBA45F0BE8, int: -1726180870236146000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a623a0, hex: 0000000000000000, int: 0},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f92793bf18d, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf197, hex: 3230323430393133303435363131, cstr: 20240913045611},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1a5, hex: 61C3D32B65955E40, int: 4638308927997723489},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1ad, hex: E5ED08A7056B4240, int: 4630381038911679973},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62598, hex: 9B00000000000000, int: 155},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a625d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f92793bf1b7, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f92793bf1bf, hex: 3230323430393133303634313032, cstr: 20240913064102},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f92793bf1cd, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92793bc680 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=2 list_head=0x7f92793bf110 latest_compact_node=(nil) max_trans_version={val:1726180870236146000, v:0} max_trans_id=125110814 max_elr_trans_version={val:1726180870236146000, v:0} max_elr_trans_id=125110814 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=2 max_modify_scn={val:1726180868176189001, v:0} min_modify_scn={val:1726180868176189001, v:0}}, version_iter:this=0x7f92793bc710 trans_version={val:1726180870236146000, v:0} scn={val:1726180868176189001, v:0} tx_id={txid:125110814} prev=(nil) next=0x7f92793bf110 modify_count=0 acc_checksum=1039313806 version=1726178586914729 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726180865533580, multi_version_iter:NULL, max_committed_trans_version:1726180870236146000, cur_trans_version:{val:1726180870236146000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:23, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:20, cur_pos:0, count:0}}, {ObPartitionMinorRowMergeIter:{tablet_id:{id:200240}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1711680543204904, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287d32320, local_datums:0x7f9287d32320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d325d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, store_ctx:{this:0x7f92bc912e98, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92bc912e98, limit_param:null, stmt_allocator:0x7f92bc913558, allocator:0x7f92bc913558, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726167149847575000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9287d32320, local_datums:0x7f9287d32320}, datums:[col_id=0:{len: 9, flag: 0, null: 0, ptr: 0x7f9288c0ac0c, hex: 313030303030333830, cstr: 100000380},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32368, hex: 280663721F6C0BE8, int: -1726167149847575000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d323a0, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=3:{len: 10, flag: 0, null: 0, ptr: 0x7f9288c0ac15, hex: 716974615F636C735F62, cstr: qita_cls_b},col_id=4:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac1f, hex: 3230323430393133303235303532, cstr: 20240913025052},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac2d, hex: 9A081B9E5E955E40, int: 4638308899850225818},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac35, hex: F35487DC0C6B4240, int: 4630381069873927411},col_id=7:null,col_id=8:null,col_id=9:null,col_id=10:null,col_id=11:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d32598, hex: 0200000000000000, int: 2},col_id=12:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d325d0, hex: 0000000000000000, int: 0},col_id=13:null,col_id=14:{len: 8, flag: 0, null: 0, ptr: 0x7f9288c0ac3f, hex: 726F636B65746D71, int: 8173316877689515890, cstr: rocketmq},col_id=15:null,col_id=16:null,col_id=17:null,col_id=18:{len: 14, flag: 0, null: 0, ptr: 0x7f9288c0ac47, hex: 3230323430393133303235323238, cstr: 20240913025228},col_id=19:{len: 24, flag: 0, null: 0, ptr: 0x7f9288c0ac55, hex: 70726F766973696F6E616C5F4D5441774D4441774D7A6777, cstr: provisional_MTAwMDAwMzgw},]}, key:{"VARCHAR":"100000380", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f929843d478 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=8 list_head=0x7f9288c0ab98 latest_compact_node=(nil) max_trans_version={val:1726167149847575000, v:0} max_trans_id=125008696 max_elr_trans_version={val:1726167149847575000, v:0} max_elr_trans_id=125008696 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=8 max_modify_scn={val:1726167149526125001, v:0} min_modify_scn={val:1726164212082486001, v:0}}, version_iter:this=0x7f9288c07c00 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f928f5fd820 next=0x7f9288c0ab98 modify_count=0 acc_checksum=1039313806 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:55} seq_no=1726167148531662, multi_version_iter:this=0x7f9288c0ab98 trans_version={val:1726167149847575000, v:0} scn={val:1726167149526125001, v:0} tx_id={txid:125008696} prev=0x7f9288c07c00 next=(nil) modify_count=1 acc_checksum=3811355001 version=1726166382297503 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:143} seq_no=1726167149130608, max_committed_trans_version:1726167149847575000, cur_trans_version:{val:1726167149847575000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:80, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337670636210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:20, cur_pos:0, count:0}}]) [2024-09-26 16:01:10.659575] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=74][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:10.659578] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=2] partition merge iter row count(i=0, row_count=23, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a71fc0, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726170015675149000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201402567}, this:0x7f92a1a71fc0, timestamp:1727334201402567, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:6, clock:394264576}, host:0x7f92ba380030, arena_handle:{allocated:35651584}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726170025270220003, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405011, mt_stat_.ready_for_flush_time:1727334209405042, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.659592] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=9] partition merge iter row count(i=1, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, table={ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337670636210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.659601] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=7][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:10.659614] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:10.659618] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:10.659620] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337670658972, end_time=1727337670659620, runtime=648, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C759-0-0, dag_ret:0, dag_status:2, start_time:1727337670636502, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.659628] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C759-0-0, dag_ret:0, dag_status:2, start_time:1727337670636502, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:10.659636] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=23133, dag_cnt=40, dag_cnts_[dag.get_type()]=40, &dag=0x7f9264c3a560, dag={ObIDag:{this:0x7f9264c3a560, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C759-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670636502, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6659188001345271846}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200240}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726159999077775000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.659644] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C759-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670636499, task_id:YB42C0A8002D-0006230056D8C759-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200240;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.659678] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337670636483, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.659687] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:0, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.659701] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.659704] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.659710] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:0, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.659729] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:1}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:0, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:0, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1}, table_count:1, [{i:0, table_key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}, merge_scn:{val:1726241621541056000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670654665, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.04ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.659754] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C761-0-0, dag_ret:0, dag_status:2, start_time:1727337670654651, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f480, output_block_cnt_arr:0x7f9264b1f488, concurrent_cnt:1, estimate_row_cnt:3, estimate_occupy_size:6291456, latest_update_ts:1727337670654665, estimated_finish_time:1727337687659757}) [2024-09-26 16:01:10.659761] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3d2a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C761-0-0, dag_ret:0, dag_status:2, start_time:1727337670654651, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:01:10.659766] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=5] task finish process(ret=0, start_time=1727337670654663, end_time=1727337670659766, runtime=5103, *this={this:0x7f9264c3d2a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C761-0-0, dag_ret:0, dag_status:2, start_time:1727337670654651, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:01:10.659789] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=25] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847408}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:01:10.659799] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.659814] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337670636463, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.659825] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828761605424, storage_schema:{this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828761605424, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.659833] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=5] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.659835] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.659842] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:200004}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1710828761605424, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.659857] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:200004}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828761605424, storage_schema:{this:0x7f92665a7330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:3, index_type:0, index_status:1, row_store_type:1, schema_version:1710828761605424, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:200004}, table_count:1, [{i:0, table_key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8621da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670654541, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.30ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.659874] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C760-0-0, dag_ret:0, dag_status:2, start_time:1727337670654526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92665a7460, output_block_cnt_arr:0x7f92665a7468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337670654541, estimated_finish_time:1727337687659877}) [2024-09-26 16:01:10.659880] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c3cd70, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C760-0-0, dag_ret:0, dag_status:2, start_time:1727337670654526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.659884] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=4] task finish process(ret=0, start_time=1727337670654539, end_time=1727337670659884, runtime=5345, *this={this:0x7f9264c3cd70, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C760-0-0, dag_ret:0, dag_status:2, start_time:1727337670654526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.659922] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=5] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.659927] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=4] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.659933] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670659931, task_id:YB42C0A8002D-0006230056D8C763-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.659937] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=4] schedule one task(task=0x7f9264c3dd00, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.659961] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.659968] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.659970] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.659975] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.660007] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=200004, range=0x7f92a9cd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685ea060, count:3, allocator:0x7f92a31f67d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92685ea320, local_datums:0x7f92685ea320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:10.659979] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8632d40, is_inited:true, wash_score:1727337670636240000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[816017-1849-0-2305843650239590400-2](offset=2027520,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:444}, data_tablet_id:{id:444}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1509715702, row_count:4}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8633440, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844406153818112-0](offset=2011136,size=1301,type=2,seq=0), type:"MAJOR", tablet_id:{id:444}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077198228842001}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843949813544213-0](offset=2012437,size=876,type=2,seq=0), type:"MINOR", tablet_id:{id:444}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726177998310996000, upper_trans_version:1726177998310996000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337670636579, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bce490, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209461185}, this:0x7f92a3bce490, timestamp:1727334209461185, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:54, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277298911712000, v:0}, rec_scn:{val:1726241718316746000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697998848-2](offset=2019328,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161889100-0](offset=2019660,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881365998-0](offset=2019822,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447886032-0](offset=2020048,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a775b0, 0x7f92a3bce490, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:10.660088] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=108] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:444}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.660099] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=6] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.660229] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=44][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:10.660273] ERROR issue_dba_error (ob_log.cpp:1875) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.660277] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.660285] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:10.660287] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:10.660289] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:3, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b0070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b00a8, hex: C852A9F488570BE8, int: -1726189786434547000},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92665b00e0, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:19071, column_count:8, rowkey_column_count:3, has_column_checksum:0, row_count:218, row_store_type:0, opt:0, var_column_count:0, row_offset:15549, original_length:16361, max_merged_trans_version:1726190661653507000, data_length:16361, data_zlength:5996, data_checksum:2604580500, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92684de030, buf_size:5996, data_size:16361, row_count:218, column_count:8, max_merged_trans_version:1726190661653507000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:83488362, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16361}) [2024-09-26 16:01:10.660300] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=11][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:10.660302] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:10.660304] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:10.660306] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:10.660307] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92685ea320, local_datums:0x7f92685ea320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92685ea320, local_datums:0x7f92685ea320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f92a31f6118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31f6118, limit_param:null, stmt_allocator:0x7f92a31f67d8, allocator:0x7f92a31f67d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:-7378697629483794432, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92685ea320, local_datums:0x7f92685ea320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337670636463, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:01:10.660368] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=61][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:10.660370] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:200004}, iter_end:false, schema_rowkey_column_cnt:1, schema_version:1710828761605424, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92685ea320, local_datums:0x7f92685ea320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, store_ctx:{this:0x7f92a31f6118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92a31f6118, limit_param:null, stmt_allocator:0x7f92a31f67d8, allocator:0x7f92a31f67d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:-7378697629483794432, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:1, ghost:0, last:0, reserved:0, flag:8}, snapshot_version:1726190661653507000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92685ea320, local_datums:0x7f92685ea320}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea330, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea368, hex: C852A9F488570BE8, int: -1726189786434547000},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea3a0, hex: 0000000000000000, int: 0},col_id=3:{len: 12, flag: 0, null: 0, ptr: 0x7f927b61a825, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea410, hex: 460B000000000000, int: 2886},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea448, hex: 523F190000000000, int: 1654610},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92685ea480, hex: DA90E36600000000, int: 1726189786},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927b61a83b, hex: 806A02E1F5210600, int: 1726189786000000},]}, key:{"BIGINT":1}, value_iter:{value:{this=0x7f927d8040a8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=6 list_head=0x7f927d822598 latest_compact_node=0x7f927b61bc08 max_trans_version={val:1726190661653507000, v:0} max_trans_id=125184312 max_elr_trans_version={val:1726190661653507000, v:0} max_elr_trans_id=125184312 latest_compact_ts=1727334202857703 last_compact_cnt=955 total_trans_node_cnt=1104 max_modify_scn={val:1726190661653507000, v:0} min_modify_scn={val:1726178588939821000, v:0}}, version_iter:this=0x7f927b61cfb8 trans_version={val:1726190657645080000, v:0} scn={val:1726190657645080000, v:0} tx_id={txid:125184285} prev=0x7f927b812588 next=0x7f927d822598 modify_count=1102 acc_checksum=2262728325 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726190657531709, multi_version_iter:this=0x7f927d8213e0 trans_version={val:1726189782404675000, v:0} scn={val:1726189782404675000, v:0} tx_id={txid:125178484} prev=0x7f927b80eb20 next=0x7f927b61a7b0 modify_count=886 acc_checksum=3974323294 version=1726178588590324 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:59} seq_no=1726189782220119, max_committed_trans_version:1726190661653507000, cur_trans_version:{val:1726189786434547000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:218, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337670636463, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:01:10.660405] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=35][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:10.660407] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=1] partition merge iter row count(i=0, row_count=218, ghost_row_count=0, pkey={tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a74ea0, key:{tablet_id:{id:200004}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201766848}, this:0x7f92a1a74ea0, timestamp:1727334201766848, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828761605424, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:11, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178588939821000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405168, mt_stat_.create_flush_dag_time:1727337670636463, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.660418] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:10.660427] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:10.660430] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:10.660432] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337670659948, end_time=1727337670660432, runtime=484, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C760-0-0, dag_ret:0, dag_status:2, start_time:1727337670654526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.660438] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=6][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C760-0-0, dag_ret:0, dag_status:2, start_time:1727337670654526, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:10.660445] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=5919, dag_cnt=39, dag_cnts_[dag.get_type()]=39, &dag=0x7f9264c3c9b0, dag={ObIDag:{this:0x7f9264c3c9b0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C760-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670654526, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6809498982591245614}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:200004}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.660454] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C760-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337670654520, task_id:YB42C0A8002D-0006230056D8C760-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=200004;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.660515] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=1, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:8}, cols_desc:{ObIArray:[column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:5, allocator:0x7f92691ac7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:10.660916] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=12][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:10.660928] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:1152921504606847409}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=1726077600415496000, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1725991205062061000) [2024-09-26 16:01:10.660943] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=9] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.660953] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.660972] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=18][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.660978] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=6][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:10.660980] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:10.660982] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92665fc178, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc0a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92665fc189, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc118, hex: 60D2640CCC290BE8, int: -1726240075808452000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-20032, column_count:8, rowkey_column_count:5, has_column_checksum:0, row_count:158, row_store_type:0, opt:0, var_column_count:0, row_offset:15849, original_length:16421, max_merged_trans_version:1726241621541056000, data_length:16421, data_zlength:16421, data_checksum:3599888316, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f92b6c04090, buf_size:16421, data_size:16421, row_count:158, column_count:8, max_merged_trans_version:1726241621541056000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:1714289658, row_count_delta:2, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16421}) [2024-09-26 16:01:10.660994] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:10.660996] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=1][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:10.660997] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:10.660998] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:01:10.661000] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=2][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:10.661001] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=1][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:10.661003] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f92691ac118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691ac118, limit_param:null, stmt_allocator:0x7f92691ac7d8, allocator:0x7f92691ac7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337670636483, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:01:10.661057] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=54][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:10.661059] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=2][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:0, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, store_ctx:{this:0x7f92691ac118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691ac118, limit_param:null, stmt_allocator:0x7f92691ac7d8, allocator:0x7f92691ac7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241621541056000, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"UPDATE", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:1726241621541056000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92684b4360, local_datums:0x7f92684b4360}, datums:[col_id=0:{len: 17, flag: 0, null: 0, ptr: 0x7f92836ac76c, hex: 5F5F616C6C5F676C6F62616C5F73746174, cstr: __all_global_stat},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b43a8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 15, flag: 0, null: 0, ptr: 0x7f92836ac77e, hex: 736E617073686F745F67635F73636E, cstr: snapshot_gc_scn},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4418, hex: 60D2640CCC290BE8, int: -1726240075808452000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92684b4450, hex: 0000000000000000, int: 0},col_id=5:NOP,col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f92836ac78d, hex: D3D8839601220600, int: 1726240075798739},col_id=7:{len: 19, flag: 0, null: 0, ptr: 0x7f92836ac795, hex: 31373236323430303735373938373339303030, cstr: 1726240075798739000},]}, key:{"VARCHAR":"__all_global_stat", collation:"utf8mb4_general_ci"},{"BIGINT":1},{"VARCHAR":"snapshot_gc_scn", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f92836040e8 latch_=unlocked flag=3 first_dml=UPDATE last_dml=UPDATE update_since_compact=9 list_head=0x7f927badc270 latest_compact_node=0x7f927c0d0780 max_trans_version={val:1726241621541056000, v:0} max_trans_id=125583518 max_elr_trans_version={val:1726241621541056000, v:0} max_elr_trans_id=125583518 latest_compact_ts=1727334209390483 last_compact_cnt=5829 total_trans_node_cnt=6273 max_modify_scn={val:1726241621541056000, v:0} min_modify_scn={val:1726178588939821001, v:0}}, version_iter:NULL, multi_version_iter:this=0x7f927c0cc778 trans_version={val:1726240065789266000, v:0} scn={val:1726240065789266000, v:0} tx_id={txid:125571316} prev=0x7f927c0cbb78 next=0x7f92836ac6f8 modify_count=5617 acc_checksum=2471975035 version=1726211869375609 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:2, buf_len:88} seq_no=1726240065537260, max_committed_trans_version:1726241621541056000, cur_trans_version:{val:1726240075808452000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241621541056000, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:158, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337670636483, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:01:10.661092] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=32][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:10.661094] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=1] partition merge iter row count(i=0, row_count=158, ghost_row_count=0, pkey={tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, table={ObITable:{this:0x7f92a1a76610, key:{tablet_id:{id:1}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782990}, this:0x7f92a1a76610, timestamp:1727334201782990, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:14, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241621541056000, v:0}, rec_scn:{val:1726178588939821001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209506202, mt_stat_.create_flush_dag_time:1727337670636483, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.661105] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:10.661113] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:10.661116] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:10.661118] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337670660480, end_time=1727337670661118, runtime=638, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C761-0-0, dag_ret:0, dag_status:2, start_time:1727337670654651, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}) [2024-09-26 16:01:10.661123] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=5][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C761-0-0, dag_ret:0, dag_status:2, start_time:1727337670654651, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:10.661130] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=6479, dag_cnt=38, dag_cnts_[dag.get_type()]=38, &dag=0x7f9264c3cee0, dag={ObIDag:{this:0x7f9264c3cee0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C761-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670654651, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6815944285528637599}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241621541056000, v:0}}}}) [2024-09-26 16:01:10.661136] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C761-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670654645, task_id:YB42C0A8002D-0006230056D8C761-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.661160] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:10.661164] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=2][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.661193] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337670636562, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.661211] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=18] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753247280, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828753247280, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.661223] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.661227] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.661232] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:412}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753247280, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.661250] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:412}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753247280, storage_schema:{this:0x7f9266505330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753247280, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:412}, table_count:1, [{i:0, table_key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670657522, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.71ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.661272] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C762-0-0, dag_ret:0, dag_status:2, start_time:1727337670657505, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9266505470, output_block_cnt_arr:0x7f9266505478, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:6291456, latest_update_ts:1727337670657522, estimated_finish_time:1727337687661275}) [2024-09-26 16:01:10.661277] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c3d7d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C762-0-0, dag_ret:0, dag_status:2, start_time:1727337670657505, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.661282] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=5] task finish process(ret=0, start_time=1727337670657519, end_time=1727337670661282, runtime=3763, *this={this:0x7f9264c3d7d0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C762-0-0, dag_ret:0, dag_status:2, start_time:1727337670657505, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.661330] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=3] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.661337] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337670661335, task_id:YB42C0A8002D-0006230056D8C764-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.661340] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=2] schedule one task(task=0x7f9264c3e230, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.661346] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670661345, task_id:YB42C0A8002D-0006230056D8C765-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.661348] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=2] schedule one task(task=0x7f9264c3e760, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.661435] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=412, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:7}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d32060, count:4, allocator:0x7f92636307d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f9287d32350, local_datums:0x7f9287d32350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:01:10.661483] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=18][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:10.661528] ERROR issue_dba_error (ob_log.cpp:1875) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.661528] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337670636398, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.661531] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.661539] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:10.661538] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=9] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:01:10.661541] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:10.661543] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=4] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92728f1480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.661543] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:4, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 23, flag: 0, null: 0, ptr: 0x7f9264a62140, hex: 6865617274626561745F736572766963655F65706F6368, cstr: heartbeat_service_epoch},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a620e0, hex: 8808AF9C9C5C0BE8, int: -1726184204453083000},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264a62118, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-7627, column_count:7, rowkey_column_count:4, has_column_checksum:0, row_count:1, row_store_type:0, opt:5, var_column_count:0, row_offset:136, original_length:80, max_merged_trans_version:1726184204453083000, data_length:80, data_zlength:80, data_checksum:1549639079, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f92b6804090, buf_size:80, data_size:80, row_count:1, column_count:7, max_merged_trans_version:1726184204453083000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:37835160, row_count_delta:0, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:80}) [2024-09-26 16:01:10.661555] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=12][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:10.661557] WDIAG [STORAGE] close (ob_macro_block_writer.cpp:718) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=2][errcode=-4184] macro block writer fail to build current micro block.(ret=-4184) [2024-09-26 16:01:10.661559] WDIAG [STORAGE] close (ob_partition_merger.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=2][errcode=-4184] Failed to close macro block writer(ret=-4184) [2024-09-26 16:01:10.661561] WDIAG [STORAGE] close (ob_partition_merger.cpp:1007) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=1][errcode=-4184] Failed to finish merge for partition merger(ret=-4184) [2024-09-26 16:01:10.661562] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1178) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=1][errcode=-4184] failed to close partition merger(ret=-4184) [2024-09-26 16:01:10.661564] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=1] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75e40, key:{tablet_id:{id:412}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201782845}, this:0x7f92a1a75e40, timestamp:1727334201782845, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753247280, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:13, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178589204684000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405197, mt_stat_.create_flush_dag_time:1727337670636562, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.661579] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=12][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:10.661587] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:10.661592] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:10.661593] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337670661389, end_time=1727337670661593, runtime=204, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C762-0-0, dag_ret:0, dag_status:2, start_time:1727337670657505, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.661602] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=8][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C762-0-0, dag_ret:0, dag_status:2, start_time:1727337670657505, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:10.661609] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=4] dag finished(dag_ret=-4184, runtime=4104, dag_cnt=37, dag_cnts_[dag.get_type()]=37, &dag=0x7f9264c3d410, dag={ObIDag:{this:0x7f9264c3d410, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C762-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670657505, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-4182959058374342994}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:412}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726176799210367000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.661617] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C762-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670657484, task_id:YB42C0A8002D-0006230056D8C762-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=412;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.661673] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=7] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3214, total_rows=22498) [2024-09-26 16:01:10.661749] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.661753] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.661756] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.661764] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=4] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.661768] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:331}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.661776] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241465393532000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.661973] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.661980] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.661982] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.661987] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.661992] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:309}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.661999] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241316899724000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.662178] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.662193] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=11] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847408}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.662218] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=19] succeed to build merge ctx(tablet_id={id:1152921504606847408}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92728f1480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847408}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670654420, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=7.77ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.662344] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75E-0-0, dag_ret:0, dag_status:2, start_time:1727337670653786, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f15d0, output_block_cnt_arr:0x7f92728f15d8, concurrent_cnt:1, estimate_row_cnt:35098, estimate_occupy_size:18874368, latest_update_ts:1727337670654420, estimated_finish_time:1727337687662353}) [2024-09-26 16:01:10.662354] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264c3c310, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75E-0-0, dag_ret:0, dag_status:2, start_time:1727337670653786, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.662360] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=5] task finish process(ret=0, start_time=1727337670654417, end_time=1727337670662359, runtime=7942, *this={this:0x7f9264c3c310, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75E-0-0, dag_ret:0, dag_status:2, start_time:1727337670653786, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.662420] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=2] schedule one task(task=0x7f9264d76030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.662428] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337670662426, task_id:YB42C0A8002D-0006230056D8C766-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.662432] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=4] schedule one task(task=0x7f9264c3ec90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.662530] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=1152921504606847408, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5e060, count:5, allocator:0x7f92691aa7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:01:10.662892] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=19][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:10.662947] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.662958] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=10][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.662966] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:10.662996] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=30][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:10.662999] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9264d58178, hex: 30323232362D32312D373825, cstr: 02226-21-78%},idx=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9264d58184, hex: 3230323430393133303435343039, cstr: 20240913045409},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d580e0, hex: CD7F500300000000, int: 55607245},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d58118, hex: E8EC05CCA0600BE8, int: -1726179788432479000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9264d58150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-11869, column_count:5, rowkey_column_count:5, has_column_checksum:0, row_count:238, row_store_type:0, opt:5, var_column_count:0, row_offset:15429, original_length:16321, max_merged_trans_version:1726190643638412000, data_length:16321, data_zlength:5388, data_checksum:152456761, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f9264d5a030, buf_size:5388, data_size:16321, row_count:238, column_count:5, max_merged_trans_version:1726190643638412000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2114797869, row_count_delta:238, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:16321}) [2024-09-26 16:01:10.663011] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=12][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:10.663013] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=2][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:10.663014] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:10.663016] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:10.663017] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92691aa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691aa118, limit_param:null, stmt_allocator:0x7f92691aa7d8, allocator:0x7f92691aa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337670636398, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}) [2024-09-26 16:01:10.663072] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=55][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:10.663073] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:1152921504606847408}, iter_end:false, schema_rowkey_column_cnt:3, schema_version:1725120002010696, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},]}, store_ctx:{this:0x7f92691aa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691aa118, limit_param:null, stmt_allocator:0x7f92691aa7d8, allocator:0x7f92691aa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:1, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726179788432479000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263b5e360, local_datums:0x7f9263b5e360}, datums:[col_id=0:{len: 12, flag: 0, null: 0, ptr: 0x7f9278d4dc7c, hex: 30323232362D32312D373825, cstr: 02226-21-78%},col_id=1:{len: 14, flag: 0, null: 0, ptr: 0x7f9278d4dc88, hex: 3230323430393133303435343039, cstr: 20240913045409},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e3e0, hex: CD7F500300000000, int: 55607245},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e418, hex: E8EC05CCA0600BE8, int: -1726179788432479000},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9263b5e450, hex: 0000000000000000, int: 0},]}, key:{"VARCHAR":"02226-21-78%", collation:"utf8mb4_general_ci"},{"VARCHAR":"20240913045409", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":55607245}, value_iter:{value:{this=0x7f9278d4db78 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f9278d4dc08 latest_compact_node=(nil) max_trans_version={val:1726179788432479000, v:0} max_trans_id=125103683 max_elr_trans_version={val:1726179788432479000, v:0} max_elr_trans_id=125103683 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726179788199545001, v:0} min_modify_scn={val:1726179788199545001, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726179788432479000, cur_trans_version:{val:1726179788432479000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:238, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337670636398, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:5, cur_pos:0, count:0}}]) [2024-09-26 16:01:10.663108] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=35][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:10.663111] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=2] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a746d0, key:{tablet_id:{id:1152921504606847408}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201765957}, this:0x7f92a1a746d0, timestamp:1727334201765957, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:10, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405135, mt_stat_.create_flush_dag_time:1727337670636398, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.663124] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:10.663142] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:10.663147] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=5][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:10.663149] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337670662481, end_time=1727337670663149, runtime=668, *this={this:0x7f9264d76030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75E-0-0, dag_ret:0, dag_status:2, start_time:1727337670653786, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.663156] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264d76030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75E-0-0, dag_ret:0, dag_status:2, start_time:1727337670653786, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:10.663170] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=10] dag finished(dag_ret=-4184, runtime=9384, dag_cnt=36, dag_cnts_[dag.get_type()]=36, &dag=0x7f9264c3bf50, dag={ObIDag:{this:0x7f9264c3bf50, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670653786, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8791616776033068452}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847408}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.663206] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75E-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670653779, task_id:YB42C0A8002D-0006230056D8C75E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847408;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.663247] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.663251] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.663253] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.663258] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.663261] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:458}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.663269] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.663529] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670663526, task_id:YB42C0A8002D-0006230056D8C767-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.663538] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=9] schedule one task(task=0x7f9264c3f1c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.663565] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.663570] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.663572] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.663576] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.663579] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:487}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.663586] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.663823] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337670636579, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.663868] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=45] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753327496, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753327496, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.663885] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.663888] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.663903] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=11] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:444}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753327496, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.663925] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:444}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753327496, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753327496, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:444}, table_count:1, [{i:0, table_key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8632d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670659955, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.94ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.663961] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=28] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C763-0-0, dag_ret:0, dag_status:2, start_time:1727337670659937, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3490, output_block_cnt_arr:0x7f9287db3498, concurrent_cnt:1, estimate_row_cnt:4, estimate_occupy_size:6291456, latest_update_ts:1727337670659955, estimated_finish_time:1727337687663964}) [2024-09-26 16:01:10.663968] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c3dd00, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C763-0-0, dag_ret:0, dag_status:2, start_time:1727337670659937, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.663973] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=5] task finish process(ret=0, start_time=1727337670659952, end_time=1727337670663973, runtime=4021, *this={this:0x7f9264c3dd00, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C763-0-0, dag_ret:0, dag_status:2, start_time:1727337670659937, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.663993] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=4] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.664177] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=444, range=0x7f92a9bd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=19 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:6, allocator:0x7f92691aa7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:01:10.664375] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=18][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:10.664416] ERROR issue_dba_error (ob_log.cpp:1875) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.664420] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.664429] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=8][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:10.664432] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:10.664434] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc0a8, hex: 0300000000000000, int: 3},idx=2:{len: 12, flag: 0, null: 0, ptr: 0x7f92665fc1b0, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc118, hex: 420B000000000000, int: 2882},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc150, hex: 288E681FE7280BE8, int: -1726241059036951000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92665fc188, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:24981, column_count:9, rowkey_column_count:6, has_column_checksum:0, row_count:207, row_store_type:0, opt:0, var_column_count:0, row_offset:15610, original_length:16378, max_merged_trans_version:1726241059036951000, data_length:16378, data_zlength:16378, data_checksum:1574137226, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926f404090, buf_size:16378, data_size:16378, row_count:207, column_count:9, max_merged_trans_version:1726241059036951000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:666643265, row_count_delta:3, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16378}) [2024-09-26 16:01:10.664459] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=25][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:10.664462] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=3][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:10.664464] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:10.664465] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=1][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:01:10.664475] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=9][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:10.664477] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:10.664478] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=1][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f04b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f04b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f92691aa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691aa118, limit_param:null, stmt_allocator:0x7f92691aa7d8, allocator:0x7f92691aa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f04b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337670636579, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}) [2024-09-26 16:01:10.664541] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=62][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:10.664542] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:444}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753327496, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f04b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, store_ctx:{this:0x7f92691aa118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f92691aa118, limit_param:null, stmt_allocator:0x7f92691aa7d8, allocator:0x7f92691aa7d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:1, compact:1, ghost:0, last:0, reserved:0, flag:13}, snapshot_version:1726241059036951000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f92664f0390, local_datums:0x7f92664f0390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f03d8, hex: 0300000000000000, int: 3},col_id=2:{len: 12, flag: 0, null: 0, ptr: 0x7f927f43a0d6, hex: 3139322E3136382E302E3435, cstr: 192.168.0.45},col_id=3:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0448, hex: 420B000000000000, int: 2882},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0480, hex: 288E681FE7280BE8, int: -1726241059036951000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f04b8, hex: 0100000000000080, int: -9223372036854775807, num_digit0: 2147483648},col_id=6:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0e4, hex: 00451DD101220600, int: 1726241058931968},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f927f43a0ec, hex: 701A92D718D7F417, int: 1726241058885606000},col_id=8:{len: 8, flag: 0, null: 0, ptr: 0x7f92664f0560, hex: 0100000000000000, int: 1, num_digit0: 0},]}, key:{"BIGINT":1},{"BIGINT UNSIGNED":3},{"VARCHAR":"192.168.0.45", collation:"utf8mb4_general_ci"},{"BIGINT UNSIGNED":2882}, value_iter:{value:{this=0x7f927f410a18 latch_=unlocked flag=3 first_dml=DELETE last_dml=INSERT update_since_compact=6 list_head=0x7f927f43a060 latest_compact_node=0x7f9281c14450 max_trans_version={val:1726241059036951000, v:0} max_trans_id=125579112 max_elr_trans_version={val:1726241059036951000, v:0} max_elr_trans_id=125579112 latest_compact_ts=1727334208955949 last_compact_cnt=87 total_trans_node_cnt=204 max_modify_scn={val:1726241059036951000, v:0} min_modify_scn={val:1726178598794272000, v:0}}, version_iter:this=0x7f927f439de0 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f9276812788 next=0x7f927f43a060 modify_count=94 acc_checksum=3917778951 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:4, buf_len:46} seq_no=1726241058891704, multi_version_iter:this=0x7f927f43a060 trans_version={val:1726241059036951000, v:0} scn={val:1726241059036951000, v:0} tx_id={txid:125579112} prev=0x7f927f439de0 next=(nil) modify_count=95 acc_checksum=3237231201 version=1726212078303536 type=0 flag=4 snapshot_barrier=0 snapshot_barrier_flag=0 mtd={dml_flag:3, buf_len:63} seq_no=1726241058891705, max_committed_trans_version:1726241059036951000, cur_trans_version:{val:1726241059036951000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:true}, scan_state:3}, iter_row_count:207, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337670636579, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:false, row_queue:{col_cnt:9, cur_pos:0, count:0}}]) [2024-09-26 16:01:10.664580] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=37][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:10.664582] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=1] partition merge iter row count(i=0, row_count=207, ghost_row_count=0, pkey={tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a775b0, key:{tablet_id:{id:444}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201958046}, this:0x7f92a1a775b0, timestamp:1727334201958046, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753327496, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:16, clock:442499072}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178598794272000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209405214, mt_stat_.create_flush_dag_time:1727337670636579, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.664595] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=9][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:10.664616] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=2][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:10.664621] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=4][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:10.664623] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337670664130, end_time=1727337670664622, runtime=492, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C763-0-0, dag_ret:0, dag_status:2, start_time:1727337670659937, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.664630] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C763-0-0, dag_ret:0, dag_status:2, start_time:1727337670659937, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:10.664639] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=6] dag finished(dag_ret=-4184, runtime=4702, dag_cnt=35, dag_cnts_[dag.get_type()]=35, &dag=0x7f9264c3d940, dag={ObIDag:{this:0x7f9264c3d940, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C763-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670659937, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1586518267935340}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:444}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.664648] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C763-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670659931, task_id:YB42C0A8002D-0006230056D8C763-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=444;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.664813] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337670664809, task_id:YB42C0A8002D-0006230056D8C768-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.664823] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=9] schedule one task(task=0x7f9264c3f6f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.664852] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.664856] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.664858] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.664863] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.664866] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b861cf80, is_inited:true, wash_score:1727337670636240000, hold_ref_cnt:true, ref_cnt:2, version:2, length:528, tablet_addr:[816017-1849-0-2305843670640488448-2](offset=1830912,size=616,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:455}, data_tablet_id:{id:455}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:4242650745, row_count:15909}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b861d6ec, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305845137371811840-0](offset=1822720,size=1982,type=2,seq=0), type:"MAJOR", tablet_id:{id:455}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389074974000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305845058988660670-0](offset=1824702,size=1909,type=2,seq=0), type:"MINOR", tablet_id:{id:455}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178192070487000, upper_trans_version:1726178192070487000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337670636670, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bcec60, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209461810}, this:0x7f92a3bcec60, timestamp:1727334209461810, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:55, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274239678928000, v:0}, rec_scn:{val:1726241818022255001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771548160-2](offset=1826816,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843202489049421-0](offset=1827149,size=180,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881173505-0](offset=1827329,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447693539-0](offset=1827555,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a794f0, 0x7f92a3bcec60, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:10.664953] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=86] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:455}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.664961] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.665630] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337670636621, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.665641] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.665655] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.665659] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.665665] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:331}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.665683] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:331}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:331}, table_count:1, [{i:0, table_key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}, merge_scn:{val:1726241465393532000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8633ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670661742, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.92ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.665712] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C765-0-0, dag_ret:0, dag_status:2, start_time:1727337670661348, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f480, output_block_cnt_arr:0x7f9264b1f488, concurrent_cnt:1, estimate_row_cnt:45, estimate_occupy_size:0, latest_update_ts:1727337670661742, estimated_finish_time:1727337687665711}) [2024-09-26 16:01:10.665718] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c3e760, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C765-0-0, dag_ret:0, dag_status:2, start_time:1727337670661348, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:01:10.665732] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=13] task finish process(ret=0, start_time=1727337670661740, end_time=1727337670665731, runtime=3991, *this={this:0x7f9264c3e760, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C765-0-0, dag_ret:0, dag_status:2, start_time:1727337670661348, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:01:10.665759] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337670636655, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.665797] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=38] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753417144, storage_schema:{this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828753417144, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.665876] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=8] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:01:10.665881] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.665886] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:487}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753417144, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.665907] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:487}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753417144, storage_schema:{this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753417144, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:487}, table_count:1, [{i:0, table_key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670663560, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=2.32ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.666028] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=17] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=2112, total_rows=2112) [2024-09-26 16:01:10.666143] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=4] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C767-0-0, dag_ret:0, dag_status:2, start_time:1727337670663538, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263b1f4a0, output_block_cnt_arr:0x7f9263b1f4a8, concurrent_cnt:1, estimate_row_cnt:2199, estimate_occupy_size:0, latest_update_ts:1727337670663560, estimated_finish_time:1727337687666142}) [2024-09-26 16:01:10.666152] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264c3f1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C767-0-0, dag_ret:0, dag_status:2, start_time:1727337670663538, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.666158] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=6] task finish process(ret=0, start_time=1727337670663558, end_time=1727337670666158, runtime=2600, *this={this:0x7f9264c3f1c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C767-0-0, dag_ret:0, dag_status:2, start_time:1727337670663538, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.666203] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=4] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.666215] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=10] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.666270] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=331, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a68060, count:5, allocator:0x7f92691aa7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9264a68360, local_datums:0x7f9264a68360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:01:10.666374] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337670636595, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.666389] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=14] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752918752, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=9, max_schema_version_in_memtable=1710828752918752, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.666409] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=14] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.666413] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.666433] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=16] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:309}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:11, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752918752, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.666455] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:309}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752918752, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752918752, column_cnt:9, store_column_cnt:9, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:309}, table_count:1, [{i:0, table_key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}, merge_scn:{val:1726241316899724000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8631da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670661966, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.46ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.666481] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C764-0-0, dag_ret:0, dag_status:2, start_time:1727337670661340, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3490, output_block_cnt_arr:0x7f92728f3498, concurrent_cnt:1, estimate_row_cnt:6, estimate_occupy_size:6291456, latest_update_ts:1727337670661966, estimated_finish_time:1727337687666484}) [2024-09-26 16:01:10.666488] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c3e230, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C764-0-0, dag_ret:0, dag_status:2, start_time:1727337670661340, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:01:10.666494] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=6] task finish process(ret=0, start_time=1727337670661963, end_time=1727337670666494, runtime=4531, *this={this:0x7f9264c3e230, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C764-0-0, dag_ret:0, dag_status:2, start_time:1727337670661340, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:01:10.666505] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=2] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.666534] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=15][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:10.666577] ERROR issue_dba_error (ob_log.cpp:1875) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.666581] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.666588] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:10.666591] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=3][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:10.666593] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=2][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:5, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38070, hex: 0000000000000000, int: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d380a8, hex: 3501000000000000, int: 309},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d380e0, hex: 3501000000000000, int: 309},idx=3:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38118, hex: 7032CD318A310BE8, int: -1726231562555674000},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9287d38150, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-24587, column_count:14, rowkey_column_count:5, has_column_checksum:0, row_count:196, row_store_type:0, opt:0, var_column_count:0, row_offset:15642, original_length:16366, max_merged_trans_version:1726241465393532000, data_length:16366, data_zlength:16366, data_checksum:3184742917, column_checksums:null, single_version_rows:0, contain_uncommitted_rows:0, is_last_row_last_flag:0, is_valid():true}, buf:0x7f926f804090, buf_size:16366, data_size:16366, row_count:196, column_count:14, max_merged_trans_version:1726241465393532000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:2306482841, row_count_delta:7, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:false, original_size:16366}) [2024-09-26 16:01:10.666604] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=11][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:10.666606] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=2][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.666611] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=0] partition merge iter row count(i=0, row_count=196, ghost_row_count=0, pkey={tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, table={ObITable:{this:0x7f92a1a77d80, key:{tablet_id:{id:331}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202026535}, this:0x7f92a1a77d80, timestamp:1727334202026535, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:17, clock:452984832}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241465393532000, v:0}, rec_scn:{val:1726179340987317000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455979, mt_stat_.create_flush_dag_time:1727337670636621, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.666633] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670666226, end_time=1727337670666633, runtime=407, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C765-0-0, dag_ret:0, dag_status:2, start_time:1727337670661348, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}}) [2024-09-26 16:01:10.666643] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5295, dag_cnt=34, dag_cnts_[dag.get_type()]=34, &dag=0x7f9264c3e3a0, dag={ObIDag:{this:0x7f9264c3e3a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C765-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670661348, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1596830851104282047}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:331}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241465393532000, v:0}}}}) [2024-09-26 16:01:10.666652] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C765-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670661345, task_id:YB42C0A8002D-0006230056D8C765-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=331;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.666677] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670666675, task_id:YB42C0A8002D-0006230056D8C769-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.666683] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=6] schedule one task(task=0x7f9264c3fc20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.666743] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=487, range=0x7f92a9d52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:11}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:8, allocator:0x7f9269f787d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9263b5a3e0, local_datums:0x7f9263b5a3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:01:10.667086] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.667136] ERROR issue_dba_error (ob_log.cpp:1875) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.667140] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.667151] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=0] partition merge iter row count(i=0, row_count=238, ghost_row_count=0, pkey={tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a79cc0, key:{tablet_id:{id:487}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202413741}, this:0x7f92a1a79cc0, timestamp:1727334202413741, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753417144, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:21, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181810189435000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405272, mt_stat_.create_flush_dag_time:1727337670636655, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.667176] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670666682, end_time=1727337670667176, runtime=494, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C767-0-0, dag_ret:0, dag_status:2, start_time:1727337670663538, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.667194] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3656, dag_cnt=33, dag_cnts_[dag.get_type()]=33, &dag=0x7f9264c3ee00, dag={ObIDag:{this:0x7f9264c3ee00, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C767-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670663538, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5675615614797020460}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:487}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.667201] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C767-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670663526, task_id:YB42C0A8002D-0006230056D8C767-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=487;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.667223] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670667221, task_id:YB42C0A8002D-0006230056D8C76A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.667227] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=3] schedule one task(task=0x7f9264c40150, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.667249] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.667254] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.667256] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.667261] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.667264] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:501}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.667259] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=309, range=0x7f92aa9d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:9, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:11}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b56060, count:7, allocator:0x7f92691aa7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:11, datum_buffer:{capacity:32, datums:0x7f9263b563b0, local_datums:0x7f9263b563b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,]}) [2024-09-26 16:01:10.667272] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.667460] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=13][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.667493] ERROR issue_dba_error (ob_log.cpp:1875) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.667496] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.667506] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=0] partition merge iter row count(i=0, row_count=156, ghost_row_count=0, pkey={tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, table={ObITable:{this:0x7f92a1a76de0, key:{tablet_id:{id:309}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201847643}, this:0x7f92a1a76de0, timestamp:1727334201847643, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752918752, max_column_cnt:9, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:15, clock:429916160}, host:0x7f92ba380030, arena_handle:{allocated:6291456}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241316899724000, v:0}, rec_scn:{val:1726178872664269000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405017, mt_stat_.ready_for_flush_time:1727334209455953, mt_stat_.create_flush_dag_time:1727337670636595, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.667525] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670667228, end_time=1727337670667525, runtime=297, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C764-0-0, dag_ret:0, dag_status:2, start_time:1727337670661340, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}}) [2024-09-26 16:01:10.667534] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=6193, dag_cnt=32, dag_cnts_[dag.get_type()]=32, &dag=0x7f9264c3de70, dag={ObIDag:{this:0x7f9264c3de70, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C764-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670661340, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3044367941103776476}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:309}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241316899724000, v:0}}}}) [2024-09-26 16:01:10.667600] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C764-0-0] [lt=63] succeed to del sys task(removed_task={start_time:1727337670661335, task_id:YB42C0A8002D-0006230056D8C764-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=309;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.667649] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:10.667674] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.667678] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.667682] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.667687] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.667691] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726178585269925002, old_tablet_snaphsot_version=1726178585269925002, tablet={this:0x7f92b8624c80, is_inited:true, wash_score:1727337670636240000, hold_ref_cnt:true, ref_cnt:2, version:2, length:541, tablet_addr:[816017-1849-0-2305843684599267328-2](offset=1966080,size=629,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:492}, data_tablet_id:{id:492}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077600564631000, v:0}, clog_checkpoint_scn:{val:1726178585269925002, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726178585269925002, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2528622955, row_count:340306}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8625434, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305850166778642432-0](offset=1949696,size=6666,type=2,seq=0), type:"MAJOR", tablet_id:{id:492}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726077389176297000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844149529467402-0](offset=1956362,size=1062,type=2,seq=0), type:"MINOR", tablet_id:{id:492}, scn_range:{start_scn:{val:1726077600564631000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178192725955000, upper_trans_version:1726178192725955000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337670636685, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd2310, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209496637}, this:0x7f92a3bd2310, timestamp:1727334209496637, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:62, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726274246934057000, v:0}, rec_scn:{val:1726241820481609000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843365697941504-2](offset=1961984,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843215374086476-0](offset=1962316,size=192,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881308684-0](offset=1962508,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447828718-0](offset=1962734,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a78550, 0x7f92a3bd2310, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:10.667734] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=43] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:492}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.667741] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.667769] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=4] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337670636422, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.667785] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=15] update column cnt(column_cnt_=3, store_column_cnt_=3, column_cnt_=3, column_array_.count()=0) [2024-09-26 16:01:10.667818] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=32] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1725120002010696, storage_schema:{this:0x7f92684e9480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=3, max_schema_version_in_memtable=1725120002010696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.667853] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670667849, task_id:YB42C0A8002D-0006230056D8C76B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.667862] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=8] schedule one task(task=0x7f9264c40680, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.667883] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=2][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.667886] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.667888] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.667891] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.667934] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:472}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.667941] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.667946] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3223, total_rows=22561) [2024-09-26 16:01:10.667952] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.667959] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684e9480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:1152921504606847409}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:5, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:2, schema_version:1725120002010696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.667979] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:1152921504606847409}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1725120002010696, storage_schema:{this:0x7f92684e9480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1725120002010696, column_cnt:3, store_column_cnt:3, tablet_size:134217728, pctfree:0, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:6, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:1152921504606847409}, table_count:1, [{i:0, table_key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8640e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670653936, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=14.02ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.668108] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75F-0-0, dag_ret:0, dag_status:2, start_time:1727337670653793, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684e95d0, output_block_cnt_arr:0x7f92684e95d8, concurrent_cnt:1, estimate_row_cnt:32439, estimate_occupy_size:18874368, latest_update_ts:1727337670653936, estimated_finish_time:1727337687668116}) [2024-09-26 16:01:10.668118] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=9] succeed to init merge ctx(task={this:0x7f9264c3c840, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75F-0-0, dag_ret:0, dag_status:2, start_time:1727337670653793, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.668124] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=5] task finish process(ret=0, start_time=1727337670653932, end_time=1727337670668123, runtime=14191, *this={this:0x7f9264c3c840, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75F-0-0, dag_ret:0, dag_status:2, start_time:1727337670653793, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.668143] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=4] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.668197] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=1152921504606847409, range=0x7f92aaad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:3, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:5}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:5, allocator:0x7f9275dfe7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:5, datum_buffer:{capacity:32, datums:0x7f9263a38360, local_datums:0x7f9263a38360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,]}) [2024-09-26 16:01:10.668442] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.668499] ERROR issue_dba_error (ob_log.cpp:1875) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.668503] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4eba37 0xf217732 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.668512] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=0] partition merge iter row count(i=0, row_count=250, ghost_row_count=0, pkey={tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a75670, key:{tablet_id:{id:1152921504606847409}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334201777083}, this:0x7f92a1a75670, timestamp:1727334201777083, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:3, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:12, clock:421527552}, host:0x7f92ba380030, arena_handle:{allocated:18874368}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726178586905254000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405016, mt_stat_.ready_for_flush_time:1727334209405151, mt_stat_.create_flush_dag_time:1727337670636422, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.668538] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670668158, end_time=1727337670668538, runtime=380, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75F-0-0, dag_ret:0, dag_status:2, start_time:1727337670653793, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.668550] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=14756, dag_cnt=31, dag_cnts_[dag.get_type()]=31, &dag=0x7f9264c3c480, dag={ObIDag:{this:0x7f9264c3c480, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C75F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670653793, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6550230011160095889}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:1152921504606847409}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.668558] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C75F-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670653790, task_id:YB42C0A8002D-0006230056D8C75F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=1152921504606847409;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.668608] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=5] succeed to add sys task(task={start_time:1727337670668606, task_id:YB42C0A8002D-0006230056D8C76C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.668612] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=4] schedule one task(task=0x7f9264c40bb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.668640] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.668644] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.668647] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726077706339704027) [2024-09-26 16:01:10.668651] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.668654] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:342}, multi_version_start=1726077600415496000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.668661] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.668673] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=12] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337670636640, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.668691] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=17] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753357664, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=7, max_schema_version_in_memtable=1710828753357664, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.668808] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=11] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:01:10.668814] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.668822] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:458}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:9, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753357664, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.668845] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=17] succeed to build merge ctx(tablet_id={id:458}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753357664, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753357664, column_cnt:7, store_column_cnt:7, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:458}, table_count:1, [{i:0, table_key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8627b60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670663241, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.58ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.668955] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=20] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1132, total_rows=7924) [2024-09-26 16:01:10.669220] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=3] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C766-0-0, dag_ret:0, dag_status:2, start_time:1727337670662432, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b214a0, output_block_cnt_arr:0x7f9264b214a8, concurrent_cnt:1, estimate_row_cnt:14598, estimate_occupy_size:4194304, latest_update_ts:1727337670663241, estimated_finish_time:1727337687669221}) [2024-09-26 16:01:10.669234] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=13] succeed to init merge ctx(task={this:0x7f9264c3ec90, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C766-0-0, dag_ret:0, dag_status:2, start_time:1727337670662432, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.669240] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=5] task finish process(ret=0, start_time=1727337670663239, end_time=1727337670669240, runtime=6001, *this={this:0x7f9264c3ec90, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C766-0-0, dag_ret:0, dag_status:2, start_time:1727337670662432, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.669262] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=5] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.669283] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337670636685, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.669295] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753437040, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=57, max_schema_version_in_memtable=1710828753437040, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.669327] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=458, range=0x7f92aa856200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:7, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:9}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a38060, count:8, allocator:0x7f92636167d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:9, datum_buffer:{capacity:32, datums:0x7f9263a383e0, local_datums:0x7f9263a383e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,]}) [2024-09-26 16:01:10.669398] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1362, total_rows=1362) [2024-09-26 16:01:10.669402] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.669409] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:492}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:12, rowkey_column_count:11, schema_rowkey_col_cnt:9, full_stored_col_cnt:59, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753437040, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=28 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.669446] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=32] succeed to build merge ctx(tablet_id={id:492}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753437040, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753437040, column_cnt:57, store_column_cnt:57, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:9, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:24, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:492}, table_count:1, [{i:0, table_key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8624c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670667669, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.74ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.669540] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C769-0-0, dag_ret:0, dag_status:2, start_time:1727337670666683, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b234c0, output_block_cnt_arr:0x7f9264b234c8, concurrent_cnt:1, estimate_row_cnt:1362, estimate_occupy_size:2097152, latest_update_ts:1727337670667669, estimated_finish_time:1727337687669541}) [2024-09-26 16:01:10.669549] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=8] succeed to init merge ctx(task={this:0x7f9264c3fc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C769-0-0, dag_ret:0, dag_status:2, start_time:1727337670666683, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.669554] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=4] task finish process(ret=0, start_time=1727337670667668, end_time=1727337670669553, runtime=1885, *this={this:0x7f9264c3fc20, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C769-0-0, dag_ret:0, dag_status:2, start_time:1727337670666683, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.669583] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.669622] ERROR issue_dba_error (ob_log.cpp:1875) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.669625] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.669584] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337670636670, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.669636] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=0] partition merge iter row count(i=0, row_count=247, ghost_row_count=0, pkey={tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7a490, key:{tablet_id:{id:458}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202425732}, this:0x7f92a1a7a490, timestamp:1727334202425732, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753357664, max_column_cnt:7, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:22, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181801157803001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405256, mt_stat_.create_flush_dag_time:1727337670636640, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.669659] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670669279, end_time=1727337670669659, runtime=380, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C766-0-0, dag_ret:0, dag_status:2, start_time:1727337670662432, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.669641] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=57] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753332736, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=36, max_schema_version_in_memtable=1710828753332736, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.669670] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=7238, dag_cnt=30, dag_cnts_[dag.get_type()]=30, &dag=0x7f9264c3e8d0, dag={ObIDag:{this:0x7f9264c3e8d0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C766-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670662432, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2360195962081676673}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:458}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.669678] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C766-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670662426, task_id:YB42C0A8002D-0006230056D8C766-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=458;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.669711] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337670636731, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.669721] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753001832, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=21, max_schema_version_in_memtable=1710828753001832, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.669735] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.669740] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=5] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.669746] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:342}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:23, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753001832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.669780] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=28] succeed to build merge ctx(tablet_id={id:342}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753001832, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753001832, column_cnt:21, store_column_cnt:21, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:342}, table_count:1, [{i:0, table_key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670668634, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.11ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.669806] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=21] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76C-0-0, dag_ret:0, dag_status:2, start_time:1727337670668612, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515490, output_block_cnt_arr:0x7f9268515498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337670668634, estimated_finish_time:1727337687669806}) [2024-09-26 16:01:10.669813] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c40bb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76C-0-0, dag_ret:0, dag_status:2, start_time:1727337670668612, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.669818] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=5] task finish process(ret=0, start_time=1727337670668632, end_time=1727337670669818, runtime=1186, *this={this:0x7f9264c40bb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76C-0-0, dag_ret:0, dag_status:2, start_time:1727337670668612, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.669833] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=6] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.669840] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=4] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.669845] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670669843, task_id:YB42C0A8002D-0006230056D8C76D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.669849] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=3] schedule one task(task=0x7f9264c410e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.669852] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=30] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:01:10.669857] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=4] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.669866] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.669864] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:455}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:10, rowkey_column_count:9, schema_rowkey_col_cnt:7, full_stored_col_cnt:38, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753332736, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=26 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.669887] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=20][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.669889] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.669894] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.669898] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:140}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.669905] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.669896] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=25] succeed to build merge ctx(tablet_id={id:455}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753332736, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753332736, column_cnt:36, store_column_cnt:36, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:7, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:455}, table_count:1, [{i:0, table_key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670664846, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=5.02ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.670063] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337670636747, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.670074] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.670084] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.670088] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.670097] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=18] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=3892, total_rows=3892) [2024-09-26 16:01:10.670093] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:140}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.670112] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:140}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:140}, table_count:1, [{i:0, table_key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8622d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670669860, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=231us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.670134] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76D-0-0, dag_ret:0, dag_status:2, start_time:1727337670669848, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263b1f460, output_block_cnt_arr:0x7f9263b1f468, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337670669860, estimated_finish_time:1727337687670134}) [2024-09-26 16:01:10.670140] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c410e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76D-0-0, dag_ret:0, dag_status:2, start_time:1727337670669848, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.670145] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=5] task finish process(ret=0, start_time=1727337670669858, end_time=1727337670670145, runtime=287, *this={this:0x7f9264c410e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76D-0-0, dag_ret:0, dag_status:2, start_time:1727337670669848, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.670159] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=1] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.670282] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=140, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5c060, count:3, allocator:0x7f92691aa7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9263b5c320, local_datums:0x7f9263b5c320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:01:10.670290] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=5] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C768-0-0, dag_ret:0, dag_status:2, start_time:1727337670664823, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f14b0, output_block_cnt_arr:0x7f92728f14b8, concurrent_cnt:1, estimate_row_cnt:3824, estimate_occupy_size:4194304, latest_update_ts:1727337670664846, estimated_finish_time:1727337687670292}) [2024-09-26 16:01:10.670300] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=10] succeed to init merge ctx(task={this:0x7f9264c3f6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C768-0-0, dag_ret:0, dag_status:2, start_time:1727337670664823, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.670306] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=5] task finish process(ret=0, start_time=1727337670664844, end_time=1727337670670305, runtime=5461, *this={this:0x7f9264c3f6f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C768-0-0, dag_ret:0, dag_status:2, start_time:1727337670664823, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.670323] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=4] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.670355] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.670380] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=455, range=0x7f92aa352200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:36, schema_rowkey_cnt:7, rowkey_cnt:9, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:7, column_cnt:38}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b00060, count:9, allocator:0x7f92b323e7d8, capacity:9, init_cnt:9}, datum_utils:{is_oracle_mode:false, rowkey_cnt:9, col_cnt:9, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:38, datum_buffer:{capacity:38, datums:0x7f9265b01340, local_datums:0x7f9265b003f0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,]}) [2024-09-26 16:01:10.670395] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.670387] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=342, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:21, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:23}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9265b0c060, count:6, allocator:0x7f92691b27d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:23, datum_buffer:{capacity:32, datums:0x7f9265b0c390, local_datums:0x7f9265b0c390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,]}) [2024-09-26 16:01:10.670398] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.670408] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7b430, key:{tablet_id:{id:140}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501569}, this:0x7f92a1a7b430, timestamp:1727334202501569, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:24, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405394, mt_stat_.create_flush_dag_time:1727337670636747, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.670473] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670670246, end_time=1727337670670472, runtime=226, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76D-0-0, dag_ret:0, dag_status:2, start_time:1727337670669848, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.670485] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=637, dag_cnt=29, dag_cnts_[dag.get_type()]=29, &dag=0x7f9264c40d20, dag={ObIDag:{this:0x7f9264c40d20, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670669848, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:2986199386776543176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:140}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.670493] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76D-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670669843, task_id:YB42C0A8002D-0006230056D8C76D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=140;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.670507] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=15][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.670589] ERROR issue_dba_error (ob_log.cpp:1875) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=36][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.670594] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.670631] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7cba0, key:{tablet_id:{id:342}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202515705}, this:0x7f92a1a7cba0, timestamp:1727334202515705, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753001832, max_column_cnt:21, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:27, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184203327921000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405377, mt_stat_.create_flush_dag_time:1727337670636731, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.670656] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.670659] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670670337, end_time=1727337670670659, runtime=322, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76C-0-0, dag_ret:0, dag_status:2, start_time:1727337670668612, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.670671] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2059, dag_cnt=28, dag_cnts_[dag.get_type()]=28, &dag=0x7f9264c407f0, dag={ObIDag:{this:0x7f9264c407f0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670668612, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6766388540859538791}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:342}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077600415496000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1725843677180998000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.670680] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76C-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670668606, task_id:YB42C0A8002D-0006230056D8C76C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=342;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.670694] ERROR issue_dba_error (ob_log.cpp:1875) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.670697] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.670708] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=0] partition merge iter row count(i=0, row_count=140, ghost_row_count=0, pkey={tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a794f0, key:{tablet_id:{id:455}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202409405}, this:0x7f92a1a794f0, timestamp:1727334202409405, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753332736, max_column_cnt:36, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:20, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:4194304}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181812495559001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405289, mt_stat_.create_flush_dag_time:1727337670636670, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.670738] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670670335, end_time=1727337670670738, runtime=403, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C768-0-0, dag_ret:0, dag_status:2, start_time:1727337670664823, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.670750] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=5927, dag_cnt=27, dag_cnts_[dag.get_type()]=27, &dag=0x7f9264c3f330, dag={ObIDag:{this:0x7f9264c3f330, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C768-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670664823, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-612435342908088006}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:455}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.670758] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C768-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670664809, task_id:YB42C0A8002D-0006230056D8C768-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=455;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.670756] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=492, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:57, schema_rowkey_cnt:9, rowkey_cnt:11, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:9, column_cnt:59}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=22 {type:"BIGINT UNSIGNED", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=24 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a42060, count:11, allocator:0x7f92c1d9a7d8, capacity:11, init_cnt:11}, datum_utils:{is_oracle_mode:false, rowkey_cnt:11, col_cnt:11, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:59, datum_buffer:{capacity:59, datums:0x7f9263b5e050, local_datums:0x7f9263a42440}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,col_id=43:NOP,col_id=44:NOP,col_id=45:NOP,col_id=46:NOP,col_id=47:NOP,col_id=48:NOP,col_id=49:NOP,col_id=50:NOP,col_id=51:NOP,col_id=52:NOP,col_id=53:NOP,col_id=54:NOP,col_id=55:NOP,col_id=56:NOP,col_id=57:NOP,col_id=58:NOP,]}) [2024-09-26 16:01:10.670779] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670670778, task_id:YB42C0A8002D-0006230056D8C76E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.670783] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=4] schedule one task(task=0x7f9264c41610, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.670788] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670670788, task_id:YB42C0A8002D-0006230056D8C76F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.670791] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=2] schedule one task(task=0x7f9264c41b40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.670794] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337670670793, task_id:YB42C0A8002D-0006230056D8C770-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.670796] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=2] schedule one task(task=0x7f9264c42070, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.670815] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.670818] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.670821] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.670825] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.670829] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101009}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.670835] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.670855] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.670858] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.670859] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726178585269925002) [2024-09-26 16:01:10.670862] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.670865] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101010}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.670869] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.670997] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=22][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.671031] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.671034] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.671077] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=0] partition merge iter row count(i=0, row_count=61, ghost_row_count=0, pkey={tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78550, key:{tablet_id:{id:492}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202353889}, this:0x7f92a1a78550, timestamp:1727334202353889, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753437040, max_column_cnt:57, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:18, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181824981509000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405306, mt_stat_.create_flush_dag_time:1727337670636685, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.671098] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337670670707, end_time=1727337670671098, runtime=391, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C769-0-0, dag_ret:0, dag_status:2, start_time:1727337670666683, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.671108] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4424, dag_cnt=26, dag_cnts_[dag.get_type()]=26, &dag=0x7f9264c3f860, dag={ObIDag:{this:0x7f9264c3f860, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C769-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670666683, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7868818205990873505}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:492}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.671115] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C769-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670666675, task_id:YB42C0A8002D-0006230056D8C769-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=492;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.671141] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670671129, task_id:YB42C0A8002D-0006230056D8C771-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.671153] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=12] schedule one task(task=0x7f9264c425a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.671155] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337670636715, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.671165] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753360568, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753360568, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.671166] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337670636700, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.671178] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.671181] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.671177] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753441792, storage_schema:{this:0x7f9263a3f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828753441792, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.671187] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:472}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753360568, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.671203] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:472}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753360568, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753360568, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:472}, table_count:1, [{i:0, table_key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b862a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670667878, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=3.31ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.671225] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.671226] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76B-0-0, dag_ret:0, dag_status:2, start_time:1727337670667862, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f460, output_block_cnt_arr:0x7f9264b1f468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337670667878, estimated_finish_time:1727337687671226}) [2024-09-26 16:01:10.671231] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.671235] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:10.671234] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=10] finish find split level(ret=0, ret="OB_SUCCESS", top_level=4, btree_node_count=1295, total_rows=1295) [2024-09-26 16:01:10.671232] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c40680, type:15, status:2, dag:{ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76B-0-0, dag_ret:0, dag_status:2, start_time:1727337670667862, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.671238] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.671240] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.671244] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:323}, multi_version_start=1726112893187579000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.671252] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.671246] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a3f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:501}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753441792, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.671238] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=5] task finish process(ret=0, start_time=1727337670667877, end_time=1727337670671238, runtime=3361, *this={this:0x7f9264c40680, type:15, status:2, dag:{ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76B-0-0, dag_ret:0, dag_status:2, start_time:1727337670667862, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.671273] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:501}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753441792, storage_schema:{this:0x7f9263a3f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753441792, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:501}, table_count:1, [{i:0, table_key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670667242, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=4.00ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.671363] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=36] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76A-0-0, dag_ret:0, dag_status:2, start_time:1727337670667226, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a3f490, output_block_cnt_arr:0x7f9263a3f498, concurrent_cnt:1, estimate_row_cnt:1295, estimate_occupy_size:0, latest_update_ts:1727337670667242, estimated_finish_time:1727337687671363}) [2024-09-26 16:01:10.671371] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c40150, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76A-0-0, dag_ret:0, dag_status:2, start_time:1727337670667226, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.671376] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=5] task finish process(ret=0, start_time=1727337670667239, end_time=1727337670671376, runtime=4137, *this={this:0x7f9264c40150, type:15, status:2, dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76A-0-0, dag_ret:0, dag_status:2, start_time:1727337670667226, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.671400] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.671403] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.671406] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:10.671410] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.671413] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:377}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.671419] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.671474] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337670636782, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.671484] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.671494] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.671497] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.671501] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101010}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:01:10.671518] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:101010}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f92684ab330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101010}, table_count:1, [{i:0, table_key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8642d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670670851, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=649us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.671538] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=14] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76F-0-0, dag_ret:0, dag_status:2, start_time:1727337670670791, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ab470, output_block_cnt_arr:0x7f92684ab478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337670670851, estimated_finish_time:1727337687671538}) [2024-09-26 16:01:10.671544] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c41b40, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76F-0-0, dag_ret:0, dag_status:2, start_time:1727337670670791, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.671548] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=4] task finish process(ret=0, start_time=1727337670670850, end_time=1727337670671548, runtime=698, *this={this:0x7f9264c41b40, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76F-0-0, dag_ret:0, dag_status:2, start_time:1727337670670791, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.671565] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337670636762, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.671573] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752542832, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=2, max_schema_version_in_memtable=1710828752542832, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.671580] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=5] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.671583] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.671586] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101009}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:4, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752542832, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0]}) [2024-09-26 16:01:10.671600] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:101009}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752542832, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752542832, column_cnt:2, store_column_cnt:2, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101009}, table_count:1, [{i:0, table_key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8641da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670670809, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=775us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.671616] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=12] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76E-0-0, dag_ret:0, dag_status:2, start_time:1727337670670783, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9268515470, output_block_cnt_arr:0x7f9268515478, concurrent_cnt:1, estimate_row_cnt:20, estimate_occupy_size:0, latest_update_ts:1727337670670809, estimated_finish_time:1727337687671616}) [2024-09-26 16:01:10.671620] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=4] succeed to init merge ctx(task={this:0x7f9264c41610, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76E-0-0, dag_ret:0, dag_status:2, start_time:1727337670670783, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.671639] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=19] task finish process(ret=0, start_time=1727337670670808, end_time=1727337670671639, runtime=831, *this={this:0x7f9264c41610, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76E-0-0, dag_ret:0, dag_status:2, start_time:1727337670670783, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.671657] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337670636818, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.671666] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=41, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.671674] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.671677] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.671684] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=5] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:377}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:43, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.671701] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:377}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:41, store_column_cnt:41, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:377}, table_count:1, [{i:0, table_key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864df20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670671396, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=286us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.671735] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=30] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C771-0-0, dag_ret:0, dag_status:2, start_time:1727337670671153, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db3480, output_block_cnt_arr:0x7f9287db3488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337670671396, estimated_finish_time:1727337687671735}) [2024-09-26 16:01:10.671740] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c425a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C771-0-0, dag_ret:0, dag_status:2, start_time:1727337670671153, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.671762] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=4] task finish process(ret=0, start_time=1727337670671394, end_time=1727337670671744, runtime=350, *this={this:0x7f9264c425a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C771-0-0, dag_ret:0, dag_status:2, start_time:1727337670671153, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.671781] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=4] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.671788] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=4] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.671791] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=2] schedule one task(task=0x7f92685b2030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.671793] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=1] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.671797] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=2] schedule one task(task=0x7f92685b23c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.671851] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101009, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684de060, count:4, allocator:0x7f92691aa7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684de350, local_datums:0x7f92684de350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:10.671861] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=501, range=0x7f92aaa56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:8}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268470060, count:6, allocator:0x7f92636627d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9268470390, local_datums:0x7f9268470390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:10.671889] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.671886] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=472, range=0x7f92aa952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9268564060, count:3, allocator:0x7f9269f787d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f9268564320, local_datums:0x7f9268564320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:01:10.671925] ERROR issue_dba_error (ob_log.cpp:1875) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.671927] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.671927] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.671937] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7bc00, key:{tablet_id:{id:101009}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501640}, this:0x7f92a1a7bc00, timestamp:1727334202501640, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:25, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405429, mt_stat_.create_flush_dag_time:1727337670636762, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.671951] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=27][errcode=-4184] Fail to alloc block(ret=-4184, free_block_cnt_=0, total_block_cnt_=10240) [2024-09-26 16:01:10.671956] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670671806, end_time=1727337670671956, runtime=150, *this={this:0x7f92685b2030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76E-0-0, dag_ret:0, dag_status:2, start_time:1727337670670783, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.671964] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1181, dag_cnt=25, dag_cnts_[dag.get_type()]=25, &dag=0x7f9264c41250, dag={ObIDag:{this:0x7f9264c41250, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670670783, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6947063304478012270}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101009}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.671971] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76E-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670670778, task_id:YB42C0A8002D-0006230056D8C76E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101009;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.671991] ERROR issue_dba_error (ob_log.cpp:1875) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.671996] ERROR issue_dba_error (ob_log.cpp:1875) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.671995] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.672002] WDIAG [STORAGE] alloc_block (ob_macro_block_writer.cpp:1359) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=7][errcode=-4184] Fail to pre-alloc block for new macro block(ret=-4184, current_index=0, current_macro_seq=0) [2024-09-26 16:01:10.672000] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.672005] WDIAG [STORAGE] write_micro_block (ob_macro_block_writer.cpp:1137) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=2][errcode=-4184] Fail to pre-alloc block(ret=-4184) [2024-09-26 16:01:10.672009] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=0] partition merge iter row count(i=0, row_count=17, ghost_row_count=0, pkey={tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7ac60, key:{tablet_id:{id:472}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202431016}, this:0x7f92a1a7ac60, timestamp:1727334202431016, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753360568, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:23, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181832211358000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405358, mt_stat_.create_flush_dag_time:1727337670636715, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.672007] WDIAG [STORAGE] build_micro_block (ob_macro_block_writer.cpp:952) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=1][errcode=-4184] fail to write micro block (ret=-4184, micro_block_desc={last_rowkey:{datum_cnt:6, group_idx:0, hash:0, [idx=0:{len: 8, flag: 0, null: 0, ptr: 0x7f926842c070, hex: 0100000000000000, int: 1, num_digit0: 0},idx=1:{len: 8, flag: 0, null: 0, ptr: 0x7f926842c0a8, hex: 0100000000000000, int: 1, num_digit0: 0},idx=2:{len: 8, flag: 0, null: 0, ptr: 0x7f926842c0e0, hex: 1210000000000000, int: 4114},idx=3:{len: 32, flag: 0, null: 0, ptr: 0x7f926842c1b0, hex: 3930363139453939454439464245324544383430353537463339334639384537, cstr: 90619E99ED9FBE2ED840557F393F98E7},idx=4:{len: 8, flag: 0, null: 0, ptr: 0x7f926842c150, hex: 10D231DEC55E0BE8, int: -1726181828237078000},idx=5:{len: 8, flag: 0, null: 0, ptr: 0x7f926842c188, hex: 0000000000000000, int: 0},]store_rowkey:}, header:{magic:1005, version:2, header_size:64, header_checksum:-11843, column_count:8, rowkey_column_count:6, has_column_checksum:0, row_count:56, row_store_type:0, opt:5, var_column_count:0, row_offset:16991, original_length:17155, max_merged_trans_version:1726181831066590000, data_length:17155, data_zlength:17155, data_checksum:578983540, column_checksums:null, single_version_rows:1, contain_uncommitted_rows:0, is_last_row_last_flag:1, is_valid():true}, buf:0x7f926fc04090, buf_size:17155, data_size:17155, row_count:56, column_count:8, max_merged_trans_version:1726181831066590000, macro_id:[9223372036854775807](ver=0,mode=0,seq=0), block_offset:0, block_checksum:3628359197, row_count_delta:56, contain_uncommitted_row:false, can_mark_deletion:false, has_string_out_row:false, has_lob_out_row:true, is_last_row_last_flag:true, original_size:17155}) [2024-09-26 16:01:10.672021] WDIAG [STORAGE] build_micro_block (ob_data_macro_block_merge_writer.cpp:137) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=13][errcode=-4184] ObMacroBlockWriter fail to build_micro_block(ret=-4184) [2024-09-26 16:01:10.672023] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:594) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=1][errcode=-4184] Fail to build micro block, (ret=-4184) [2024-09-26 16:01:10.672024] WDIAG [STORAGE] append_row (ob_macro_block_writer.cpp:518) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=1][errcode=-4184] Fail to append row(ret=-4184) [2024-09-26 16:01:10.672026] WDIAG [STORAGE] append_row (ob_data_macro_block_merge_writer.cpp:72) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=2][errcode=-4184] ObMacroBlockWriter fail to append_row(ret=-4184) [2024-09-26 16:01:10.672017] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=101010, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:2, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:4}, cols_desc:{ObIArray:[column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b0060, count:4, allocator:0x7f92a339e7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:4, datum_buffer:{capacity:32, datums:0x7f92684b0350, local_datums:0x7f92684b0350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,]}) [2024-09-26 16:01:10.672027] WDIAG [STORAGE] inner_process (ob_partition_merger.cpp:1083) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=1][errcode=-4184] Failed to append row to macro writer(ret=-4184) [2024-09-26 16:01:10.672029] WDIAG [STORAGE] process (ob_partition_merger.cpp:279) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=2][errcode=-4184] Failed to inner append row(ret=-4184) [2024-09-26 16:01:10.672032] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=1] task finish process(ret=-4184, start_time=1727337670671802, end_time=1727337670672032, runtime=230, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76B-0-0, dag_ret:0, dag_status:2, start_time:1727337670667862, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.672041] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4179, dag_cnt=24, dag_cnts_[dag.get_type()]=24, &dag=0x7f9264c402c0, dag={ObIDag:{this:0x7f9264c402c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670667862, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8731750006000903454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:472}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.672047] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=11][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.672049] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670667849, task_id:YB42C0A8002D-0006230056D8C76B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=472;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.672075] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.672077] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.672031] WDIAG [STORAGE] merge_single_iter (ob_partition_merger.cpp:1251) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=2][errcode=-4184] Failed to process row(ret=-4184, cur_row={row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726181828237078000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9268470390, local_datums:0x7f9268470390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684703a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684703d8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470410, hex: 1210000000000000, int: 4114},col_id=3:{len: 32, flag: 0, null: 0, ptr: 0x7f928331a628, hex: 3930363139453939454439464245324544383430353537463339334639384537, cstr: 90619E99ED9FBE2ED840557F393F98E7},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470480, hex: 10D231DEC55E0BE8, int: -1726181828237078000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92684704b8, hex: 0000000000000000, int: 0},col_id=6:{len: 1027, flag: 0, null: 0, ptr: 0x7f928331a648, hex: 01840000494E5345525420494E544F205F5F77725F6163746976655F73657373696F6E5F686973746F7279202874656E616E745F69642C20636C75737465725F69642C20534E41505F49442C207376725F69702C207376725F706F72742C2073616D706C655F69642C2073657373696F6E5F69642C2073616D706C655F74696D},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470528, hex: 0200000000000000, int: 2},]}, merge_iter={ObPartitionMinorRowMergeIter:{tablet_id:{id:501}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753441792, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726181828237078000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9268470390, local_datums:0x7f9268470390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684703a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684703d8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470410, hex: 1210000000000000, int: 4114},col_id=3:{len: 32, flag: 0, null: 0, ptr: 0x7f928331a628, hex: 3930363139453939454439464245324544383430353537463339334639384537, cstr: 90619E99ED9FBE2ED840557F393F98E7},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470480, hex: 10D231DEC55E0BE8, int: -1726181828237078000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92684704b8, hex: 0000000000000000, int: 0},col_id=6:{len: 1027, flag: 0, null: 0, ptr: 0x7f928331a648, hex: 01840000494E5345525420494E544F205F5F77725F6163746976655F73657373696F6E5F686973746F7279202874656E616E745F69642C20636C75737465725F69642C20534E41505F49442C207376725F69702C207376725F706F72742C2073616D706C655F69642C2073657373696F6E5F69642C2073616D706C655F74696D},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470528, hex: 0200000000000000, int: 2},]}, store_ctx:{this:0x7f9263662118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263662118, limit_param:null, stmt_allocator:0x7f92636627d8, allocator:0x7f92636627d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726181828237078000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9268470390, local_datums:0x7f9268470390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684703a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684703d8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470410, hex: 1210000000000000, int: 4114},col_id=3:{len: 32, flag: 0, null: 0, ptr: 0x7f928331a628, hex: 3930363139453939454439464245324544383430353537463339334639384537, cstr: 90619E99ED9FBE2ED840557F393F98E7},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470480, hex: 10D231DEC55E0BE8, int: -1726181828237078000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92684704b8, hex: 0000000000000000, int: 0},col_id=6:{len: 1027, flag: 0, null: 0, ptr: 0x7f928331a648, hex: 01840000494E5345525420494E544F205F5F77725F6163746976655F73657373696F6E5F686973746F7279202874656E616E745F69642C20636C75737465725F69642C20534E41505F49442C207376725F69702C207376725F706F72742C2073616D706C655F69642C2073657373696F6E5F69642C2073616D706C655F74696D},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470528, hex: 0200000000000000, int: 2},]}, key:{"BIGINT":1},{"BIGINT":1},{"BIGINT":4114},{"VARCHAR":"90619E99ED9FBE2ED840557F393F98E7", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f928331a520 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f928331a5b0 latest_compact_node=(nil) max_trans_version={val:1726181828237078000, v:0} max_trans_id=125117896 max_elr_trans_version={val:1726181828237078000, v:0} max_elr_trans_id=125117896 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726181828237078000, v:0} min_modify_scn={val:1726181828237078000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726181828237078000, cur_trans_version:{val:1726181828237078000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:56, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337670636700, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:8, cur_pos:0, count:0}}) [2024-09-26 16:01:10.672091] WDIAG [STORAGE] merge_same_rowkey_iters (ob_partition_merger.cpp:1477) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=59][errcode=-4184] Failed to merge single merge iter(ret=-4184) [2024-09-26 16:01:10.672084] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=0] partition merge iter row count(i=0, row_count=20, ghost_row_count=0, pkey={tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7c3d0, key:{tablet_id:{id:101010}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202501677}, this:0x7f92a1a7c3d0, timestamp:1727334202501677, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752542832, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:26, clock:520093696}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726184204380808000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405446, mt_stat_.create_flush_dag_time:1727337670636782, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.672099] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670671993, end_time=1727337670672099, runtime=106, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76F-0-0, dag_ret:0, dag_status:2, start_time:1727337670670791, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.672106] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1315, dag_cnt=23, dag_cnts_[dag.get_type()]=23, &dag=0x7f9264c41780, dag={ObIDag:{this:0x7f9264c41780, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670670791, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-5843378789243662202}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101010}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.672112] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76F-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337670670788, task_id:YB42C0A8002D-0006230056D8C76F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101010;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.672092] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1153) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=1][errcode=-4184] Failed to merge iters with same rowkey(ret=-4184, rowkey_minimum_iters=[{ObPartitionMinorRowMergeIter:{tablet_id:{id:501}, iter_end:false, schema_rowkey_column_cnt:4, schema_version:1710828753441792, merge_range:{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}, curr_row_:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726181828237078000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9268470390, local_datums:0x7f9268470390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684703a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684703d8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470410, hex: 1210000000000000, int: 4114},col_id=3:{len: 32, flag: 0, null: 0, ptr: 0x7f928331a628, hex: 3930363139453939454439464245324544383430353537463339334639384537, cstr: 90619E99ED9FBE2ED840557F393F98E7},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470480, hex: 10D231DEC55E0BE8, int: -1726181828237078000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92684704b8, hex: 0000000000000000, int: 0},col_id=6:{len: 1027, flag: 0, null: 0, ptr: 0x7f928331a648, hex: 01840000494E5345525420494E544F205F5F77725F6163746976655F73657373696F6E5F686973746F7279202874656E616E745F69642C20636C75737465725F69642C20534E41505F49442C207376725F69702C207376725F706F72742C2073616D706C655F69642C2073657373696F6E5F69642C2073616D706C655F74696D},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470528, hex: 0200000000000000, int: 2},]}, store_ctx:{this:0x7f9263662118, ls_id:{id:1}, ls:null, timeout:9223372036854775807, tablet_id:{id:0}, table_iter:null, table_version:9223372036854775807, mvcc_acc_ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, tablet_stat:{ls_id:0, tablet_id:0, query_cnt:0, merge_cnt:0, scan_logical_row_cnt:0, scan_physical_row_cnt:0, scan_micro_block_cnt:0, pushdown_micro_block_cnt:0, exist_row_total_table_cnt:0, exist_row_read_table_cnt:0, insert_row_cnt:0, update_row_cnt:0, delete_row_cnt:0}, replay_log_scn:{val:4611686018427387903, v:0}, is_read_store_ctx:false}, row_iter_:{context:{is_inited:true, timeout:9223372036854775807, ls_id:{id:1}, tablet_id:{id:0}, query_flag:{scan_order:1, daily_merge:1, rmmb_optimize:1, whole_macro_scan:1, full_row:0, index_back:0, query_stat:0, sql_mode:0, read_latest:0, prewarm:0, join_type:0, use_row_cache:0, use_block_index_cache:0, use_bloomfilter_cache:0, multi_version_minor_merge:1, is_need_feedback:0, use_fuse_row_cache:0, use_fast_agg:0, iter_uncommitted_row:0, ignore_trans_stat:0, is_large_query:0, is_sstable_cut:0, skip_read_lob:0, is_lookup_for_4377:0, is_for_foreign_key_check:0, reserved:0}, sql_mode:0, store_ctx:0x7f9263662118, limit_param:null, stmt_allocator:0x7f92636627d8, allocator:0x7f92636627d8, range_allocator:null, table_scan_stat:null, out_cnt:0, trans_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, merge_scn:{val:1726241290462324999, v:0}, lob_allocator:pages=1, used=560, total=7936, lob_locator_helper:{table_id:0, ls_id:1, snapshot_version:0, rowid_version:0, rowid_project_:NULL, enable_locator_v2:true, is_inited:true}, iter_pool:null, block_row_store:null, io_callback:null}, row:{row_flag:{flag:"INSERT", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:1, uncommitted:0, shadow:0, compact:1, ghost:0, last:1, reserved:0, flag:41}, snapshot_version:1726181828237078000, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9268470390, local_datums:0x7f9268470390}, datums:[col_id=0:{len: 8, flag: 0, null: 0, ptr: 0x7f92684703a0, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=1:{len: 8, flag: 0, null: 0, ptr: 0x7f92684703d8, hex: 0100000000000000, int: 1, num_digit0: 0},col_id=2:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470410, hex: 1210000000000000, int: 4114},col_id=3:{len: 32, flag: 0, null: 0, ptr: 0x7f928331a628, hex: 3930363139453939454439464245324544383430353537463339334639384537, cstr: 90619E99ED9FBE2ED840557F393F98E7},col_id=4:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470480, hex: 10D231DEC55E0BE8, int: -1726181828237078000},col_id=5:{len: 8, flag: 0, null: 0, ptr: 0x7f92684704b8, hex: 0000000000000000, int: 0},col_id=6:{len: 1027, flag: 0, null: 0, ptr: 0x7f928331a648, hex: 01840000494E5345525420494E544F205F5F77725F6163746976655F73657373696F6E5F686973746F7279202874656E616E745F69642C20636C75737465725F69642C20534E41505F49442C207376725F69702C207376725F706F72742C2073616D706C655F69642C2073657373696F6E5F69642C2073616D706C655F74696D},col_id=7:{len: 8, flag: 0, null: 0, ptr: 0x7f9268470528, hex: 0200000000000000, int: 2},]}, key:{"BIGINT":1},{"BIGINT":1},{"BIGINT":4114},{"VARCHAR":"90619E99ED9FBE2ED840557F393F98E7", collation:"utf8mb4_general_ci"}, value_iter:{value:{this=0x7f928331a520 latch_=unlocked flag=3 first_dml=INSERT last_dml=INSERT update_since_compact=1 list_head=0x7f928331a5b0 latest_compact_node=(nil) max_trans_version={val:1726181828237078000, v:0} max_trans_id=125117896 max_elr_trans_version={val:1726181828237078000, v:0} max_elr_trans_id=125117896 latest_compact_ts=0 last_compact_cnt=0 total_trans_node_cnt=1 max_modify_scn={val:1726181828237078000, v:0} min_modify_scn={val:1726181828237078000, v:0}}, version_iter:NULL, multi_version_iter:NULL, max_committed_trans_version:1726181828237078000, cur_trans_version:{val:1726181828237078000, v:0}, is_node_compacted:false, ctx:{type:1, abs_lock_timeout:9223372036854775807, tx_lock_timeout:-1, snapshot:{version:{val:4611686018427387901, v:0}, tx_id:{txid:0}, scn:0}, tx_table_guards:{tx_table_guard:{tx_table:0x7f92a6c0a490, epoch:0, mini_cache_:}, src_tx_table_guard:{tx_table:null, epoch:-1, mini_cache_:}, transfer_start_scn:{val:18446744073709551615, v:3}}, tx_id:{txid:0}, tx_desc:NULL, tx_ctx:null, mem_ctx:null, tx_scn:0, write_flag:{is_table_api:0, is_table_lock:0, is_mds:0, is_dml_batch_opt:0, is_insert_up:0, is_write_only_index:0, is_check_row_locked:0}, handle_start_time:-1, lock_wait_start_ts:0}, merge_scn:{val:1726241290462324999, v:0}, version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:4611686018427387901}, has_multi_commit_trans:false}, scan_state:4}, iter_row_count:56, is_inited:true, is_rowkey_first_row_reused:false, table_:{ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337670636700, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, ghost_row_count:0, check_committing_trans_compacted:true, row_queue:{col_cnt:8, cur_pos:0, count:0}}]) [2024-09-26 16:01:10.672126] WDIAG [STORAGE] merge_partition (ob_partition_merger.cpp:1172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=34][errcode=-4184] Partition merge did not end normally(ret=-4184) [2024-09-26 16:01:10.672129] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=2] partition merge iter row count(i=0, row_count=56, ghost_row_count=0, pkey={tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a78d20, key:{tablet_id:{id:501}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334202354018}, this:0x7f92a1a78d20, timestamp:1727334202354018, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753441792, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:19, clock:507510784}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726181828111105000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405018, mt_stat_.ready_for_flush_time:1727334209405323, mt_stat_.create_flush_dag_time:1727337670636700, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.672140] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1539) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=8][errcode=-4184] failed to merge partition(ret=-4184) [2024-09-26 16:01:10.672149] WDIAG [STORAGE] process (ob_tablet_merge_task.cpp:1554) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=1][errcode=-4184] failed to merge(ret=-4184, ctx_->param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, idx_=0) [2024-09-26 16:01:10.672152] WDIAG [COMMON] do_work (ob_dag_scheduler.cpp:242) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=3][errcode=-4184] failed to process task(ret=-4184) [2024-09-26 16:01:10.672154] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=2] task finish process(ret=-4184, start_time=1727337670671809, end_time=1727337670672154, runtime=345, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76A-0-0, dag_ret:0, dag_status:2, start_time:1727337670667226, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.672161] WDIAG [COMMON] run1 (ob_dag_scheduler.cpp:1478) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=7][errcode=-4184] failed to do work(ret=-4184, *task_={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76A-0-0, dag_ret:0, dag_status:2, start_time:1727337670667226, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}, compat_mode=0) [2024-09-26 16:01:10.672156] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=377, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:41, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:43}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665b0060, count:5, allocator:0x7f92b33d87d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:43, datum_buffer:{capacity:43, datums:0x7f92665b12b0, local_datums:0x7f92665b0360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,col_id=31:NOP,col_id=32:NOP,col_id=33:NOP,col_id=34:NOP,col_id=35:NOP,col_id=36:NOP,col_id=37:NOP,col_id=38:NOP,col_id=39:NOP,col_id=40:NOP,col_id=41:NOP,col_id=42:NOP,]}) [2024-09-26 16:01:10.672169] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=5] dag finished(dag_ret=-4184, runtime=4943, dag_cnt=22, dag_cnts_[dag.get_type()]=22, &dag=0x7f9264c3fd90, dag={ObIDag:{this:0x7f9264c3fd90, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C76A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670667226, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7071731214611448811}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:501}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.672177] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C76A-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670667221, task_id:YB42C0A8002D-0006230056D8C76A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=501;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.672189] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.672215] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.672217] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.672224] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=0] partition merge iter row count(i=0, row_count=6, ghost_row_count=0, pkey={tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7e310, key:{tablet_id:{id:377}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204215429}, this:0x7f92a1a7e310, timestamp:1727334204215429, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:41, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:30, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207956191609000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405480, mt_stat_.create_flush_dag_time:1727337670636818, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.672240] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670672131, end_time=1727337670672240, runtime=109, *this={this:0x7f92685b23c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C771-0-0, dag_ret:0, dag_status:2, start_time:1727337670671153, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.672248] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1095, dag_cnt=21, dag_cnts_[dag.get_type()]=21, &dag=0x7f9264c421e0, dag={ObIDag:{this:0x7f9264c421e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C771-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670671153, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-2678407261265885087}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:377}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.672254] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C771-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670671129, task_id:YB42C0A8002D-0006230056D8C771-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=377;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.672291] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337670672289, task_id:YB42C0A8002D-0006230056D8C772-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.672295] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=3] schedule one task(task=0x7f9264c42ad0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.672301] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670672300, task_id:YB42C0A8002D-0006230056D8C773-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.672305] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=4] schedule one task(task=0x7f9264c43000, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.672309] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670672308, task_id:YB42C0A8002D-0006230056D8C774-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.672312] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=2] schedule one task(task=0x7f9264c43530, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.672314] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337670672314, task_id:YB42C0A8002D-0006230056D8C775-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.672316] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=2] schedule one task(task=0x7f9264c43a60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.672320] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670672319, task_id:YB42C0A8002D-0006230056D8C776-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.672322] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=2] schedule one task(task=0x7f9264c43f90, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.672333] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.672337] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.672338] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.672340] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:10.672341] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.672343] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:10.672347] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.672375] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.672377] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.672379] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101065}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.672382] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.672385] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:10.672387] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.672389] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.672350] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=1] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726077692196634026, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b8653ce0, is_inited:true, wash_score:1727337670636240000, hold_ref_cnt:true, ref_cnt:2, version:2, length:505, tablet_addr:[809012-1445-0-2305843645943099392-2](offset=503808,size=593,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101066}, data_tablet_id:{id:333}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726077692196634026, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3243687120, row_count:2118}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b8654404, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[809012-1445-0-2305844488830410752-0](offset=483328,size=1378,type=2,seq=0), type:"MAJOR", tablet_id:{id:101066}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[809012-1445-0-2305843931558405474-0](offset=484706,size=859,type=2,seq=0), type:"MINI", tablet_id:{id:101066}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337670636885, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bdd4f0, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221023266}, this:0x7f92a3bdd4f0, timestamp:1727334221023266, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:84, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265274283037001, v:0}, rec_scn:{val:1726264914351473001, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[809012-1445-0-2305843365696471040-2](offset=491520,size=332,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[809012-1445-0-2305843189602812236-0](offset=491852,size=168,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[809012-1445-0-2305843251879838196-0](offset=492020,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5020, 0x7f92a3bdd4f0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:10.672404] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=45] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101066}, multi_version_start=1726077692196634026, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.672411] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.672411] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.672414] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.672415] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:10.672419] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.672421] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:333}, multi_version_start=1726077706139416000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.672425] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.672430] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.672433] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.672435] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:10.672392] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726121601081682000, old_tablet_snaphsot_version=1726149987022722000, tablet={this:0x7f92b864eec0, is_inited:true, wash_score:1727337670636240000, hold_ref_cnt:true, ref_cnt:2, version:2, length:509, tablet_addr:[814042-5586-0-2305843650238771200-2](offset=1208320,size=597,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:297}, data_tablet_id:{id:297}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726063409221604000, v:0}, clog_checkpoint_scn:{val:1726149987022722000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726149987022722000, multi_version_start:1726121601081682000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:2361333062, row_count:24}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b864f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[814042-5586-0-2305844531780800512-0](offset=1200128,size=1418,type=2,seq=0), type:"MAJOR", tablet_id:{id:297}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063202518238000}}, minor_tables:{{i:0, addr:[814042-5586-0-2305843919747962250-0](offset=1201546,size=848,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726063409221604000, v:0}, end_scn:{val:1726121601081682000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726121557514739000, upper_trans_version:1726121557514739000}, {i:1, addr:[814042-5586-0-2305843920821704922-0](offset=1202394,size=849,type=2,seq=0), type:"MINI", tablet_id:{id:297}, scn_range:{start_scn:{val:1726121601081682000, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149603136610000, upper_trans_version:1726149603136610000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337670636836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd86d0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334220478269}, this:0x7f92a3bd86d0, timestamp:1727334220478269, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:74, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726264804632655000, v:0}, rec_scn:{val:1726264804632655000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[814042-5586-0-2305843479513817088-2](offset=1204224,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[814042-5586-0-2305843183161074102-0](offset=1204662,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843251880551000-0](offset=1204824,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[814042-5586-0-2305843216447071034-0](offset=1205050,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a1a7eae0, 0x7f92a3bd86d0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:10.672437] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.672437] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=45] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:297}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.672440] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:334}, multi_version_start=1726077706339704027, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.672443] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.672444] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.672461] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337670636801, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.672491] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=29] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752949376, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=20, max_schema_version_in_memtable=1710828752949376, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.672512] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.672515] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.672521] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:323}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:22, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752949376, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.672539] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:323}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752949376, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752949376, column_cnt:20, store_column_cnt:20, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:323}, table_count:1, [{i:0, table_key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864b040, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670671219, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.30ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.672570] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C770-0-0, dag_ret:0, dag_status:2, start_time:1727337670670796, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad480, output_block_cnt_arr:0x7f92684ad488, concurrent_cnt:1, estimate_row_cnt:205, estimate_occupy_size:0, latest_update_ts:1727337670671219, estimated_finish_time:1727337687672570}) [2024-09-26 16:01:10.672577] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c42070, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C770-0-0, dag_ret:0, dag_status:2, start_time:1727337670670796, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.672582] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2854][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=4] task finish process(ret=0, start_time=1727337670671216, end_time=1727337670672581, runtime=1365, *this={this:0x7f9264c42070, type:15, status:2, dag:{ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C770-0-0, dag_ret:0, dag_status:2, start_time:1727337670670796, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.672657] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=1] schedule one task(task=0x7f9262c04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.672643] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337670636854, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.672691] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=47] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.672705] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.672709] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.672714] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101065}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.672722] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=323, range=0x7f92aa6d4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:20, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:22}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:5, allocator:0x7f92af4d67d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:22, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,]}) [2024-09-26 16:01:10.672733] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:101065}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101065}, table_count:1, [{i:0, table_key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864bfe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670672328, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=384us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.672767] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=27] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C773-0-0, dag_ret:0, dag_status:2, start_time:1727337670672305, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f490, output_block_cnt_arr:0x7f9264b1f498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337670672328, estimated_finish_time:1727337687672767}) [2024-09-26 16:01:10.672774] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c43000, type:15, status:2, dag:{ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C773-0-0, dag_ret:0, dag_status:2, start_time:1727337670672305, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.672779] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2865][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=5] task finish process(ret=0, start_time=1727337670672325, end_time=1727337670672779, runtime=454, *this={this:0x7f9264c43000, type:15, status:2, dag:{ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C773-0-0, dag_ret:0, dag_status:2, start_time:1727337670672305, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.672985] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=29][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.673025] ERROR issue_dba_error (ob_log.cpp:1875) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.673029] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.673041] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=0] partition merge iter row count(i=0, row_count=171, ghost_row_count=0, pkey={tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7d370, key:{tablet_id:{id:323}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334203390856}, this:0x7f92a1a7d370, timestamp:1727334203390856, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752949376, max_column_cnt:20, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:28, clock:564133888}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726199007654744001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405019, mt_stat_.ready_for_flush_time:1727334209405463, mt_stat_.create_flush_dag_time:1727337670636801, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.673063] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670672671, end_time=1727337670673063, runtime=392, *this={this:0x7f9262c04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C770-0-0, dag_ret:0, dag_status:2, start_time:1727337670670796, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.673074] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2278, dag_cnt=20, dag_cnts_[dag.get_type()]=20, &dag=0x7f9264c41cb0, dag={ObIDag:{this:0x7f9264c41cb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C770-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670670796, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1841823684513746176}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:323}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726112893187579000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.673083] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2869][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C770-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670670793, task_id:YB42C0A8002D-0006230056D8C770-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=323;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.673111] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337670636870, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.673116] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337670636905, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.673122] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.673128] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=19, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.673141] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.673144] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.673149] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:333}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.673167] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.673170] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.673168] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:333}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:333}, table_count:1, [{i:0, table_key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8651da0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670672408, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=739us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.673176] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:334}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:21, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.673195] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C774-0-0, dag_ret:0, dag_status:2, start_time:1727337670672311, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263b1f490, output_block_cnt_arr:0x7f9263b1f498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337670672408, estimated_finish_time:1727337687673195}) [2024-09-26 16:01:10.673201] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c43530, type:15, status:2, dag:{ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C774-0-0, dag_ret:0, dag_status:2, start_time:1727337670672311, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.673206] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2873][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=5] task finish process(ret=0, start_time=1727337670672406, end_time=1727337670673206, runtime=800, *this={this:0x7f9264c43530, type:15, status:2, dag:{ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C774-0-0, dag_ret:0, dag_status:2, start_time:1727337670672311, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.673196] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:334}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9287db3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:19, store_column_cnt:19, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:23, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:334}, table_count:1, [{i:0, table_key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8654c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670672427, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=747us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.673250] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=17] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C776-0-0, dag_ret:0, dag_status:2, start_time:1727337670672322, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db34a0, output_block_cnt_arr:0x7f9287db34a8, concurrent_cnt:1, estimate_row_cnt:993, estimate_occupy_size:0, latest_update_ts:1727337670672427, estimated_finish_time:1727337687673250}) [2024-09-26 16:01:10.673257] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c43f90, type:15, status:2, dag:{ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C776-0-0, dag_ret:0, dag_status:2, start_time:1727337670672322, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.673262] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2868][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=5] task finish process(ret=0, start_time=1727337670672426, end_time=1727337670673262, runtime=836, *this={this:0x7f9264c43f90, type:15, status:2, dag:{ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C776-0-0, dag_ret:0, dag_status:2, start_time:1727337670672322, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.673278] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337670636885, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.673287] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=5, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.673296] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.673299] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.673303] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101066}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:7, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.673318] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101066}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:5, store_column_cnt:5, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101066}, table_count:1, [{i:0, table_key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8653ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670672333, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=969us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.673342] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C775-0-0, dag_ret:0, dag_status:2, start_time:1727337670672316, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b21490, output_block_cnt_arr:0x7f9264b21498, concurrent_cnt:1, estimate_row_cnt:182, estimate_occupy_size:0, latest_update_ts:1727337670672333, estimated_finish_time:1727337687673342}) [2024-09-26 16:01:10.673349] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c43a60, type:15, status:2, dag:{ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C775-0-0, dag_ret:0, dag_status:2, start_time:1727337670672316, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.673354] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2889][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=5] task finish process(ret=0, start_time=1727337670672331, end_time=1727337670673354, runtime=1023, *this={this:0x7f9264c43a60, type:15, status:2, dag:{ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C775-0-0, dag_ret:0, dag_status:2, start_time:1727337670672316, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.673384] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=5] schedule one task(task=0x7f9287d5e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.673389] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=4] schedule one task(task=0x7f9287d5e3c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.673392] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=2] schedule one task(task=0x7f9287d5e750, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.673395] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=2] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.673399] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670673398, task_id:YB42C0A8002D-0006230056D8C777-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.673403] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=4] schedule one task(task=0x7f9264c444c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.673440] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101065, range=0x7f92a9cd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684b4060, count:6, allocator:0x7f92a31787d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f92684b4390, local_datums:0x7f92684b4390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:01:10.673467] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=334, range=0x7f92a9c56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:19, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:21}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d38060, count:8, allocator:0x7f92636367d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:21, datum_buffer:{capacity:32, datums:0x7f9287d383e0, local_datums:0x7f9287d383e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,]}) [2024-09-26 16:01:10.673489] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.673524] ERROR issue_dba_error (ob_log.cpp:1875) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.673527] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.673536] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=0] partition merge iter row count(i=0, row_count=15, ghost_row_count=0, pkey={tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7db40, key:{tablet_id:{id:101065}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204214678}, this:0x7f92a1a7db40, timestamp:1727334204214678, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:29, clock:572522496}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405513, mt_stat_.create_flush_dag_time:1727337670636854, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.673554] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670673414, end_time=1727337670673554, runtime=140, *this={this:0x7f9287d5e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C773-0-0, dag_ret:0, dag_status:2, start_time:1727337670672305, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.673563] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1258, dag_cnt=19, dag_cnts_[dag.get_type()]=19, &dag=0x7f9264c42c40, dag={ObIDag:{this:0x7f9264c42c40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C773-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670672305, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3730526293242960203}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101065}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.673571] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2881][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C773-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670672300, task_id:YB42C0A8002D-0006230056D8C773-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101065;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.673615] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=333, range=0x7f92aabd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92685ea060, count:7, allocator:0x7f92691ac7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f92685ea3b0, local_datums:0x7f92685ea3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:01:10.673676] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=16][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.673727] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337670636836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.673741] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=13] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=16, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.673754] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.673757] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.673763] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:297}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:18, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.673784] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.673787] ERROR issue_dba_error (ob_log.cpp:1875) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=5][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.673791] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.673783] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:297}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f9264b23330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:16, store_column_cnt:16, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:297}, table_count:1, [{i:0, table_key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670672371, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.39ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.673802] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=0] partition merge iter row count(i=0, row_count=153, ghost_row_count=0, pkey={tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc57f0, key:{tablet_id:{id:334}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204276273}, this:0x7f92a3bc57f0, timestamp:1727334204276273, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:19, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:36, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207977350579001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405563, mt_stat_.create_flush_dag_time:1727337670636905, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.673808] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=20] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C772-0-0, dag_ret:0, dag_status:2, start_time:1727337670672294, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b23490, output_block_cnt_arr:0x7f9264b23498, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337670672371, estimated_finish_time:1727337687673808}) [2024-09-26 16:01:10.673815] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c42ad0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C772-0-0, dag_ret:0, dag_status:2, start_time:1727337670672294, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.673818] ERROR issue_dba_error (ob_log.cpp:1875) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.673820] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2859][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=5] task finish process(ret=0, start_time=1727337670672368, end_time=1727337670673820, runtime=1452, *this={this:0x7f9264c42ad0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C772-0-0, dag_ret:0, dag_status:2, start_time:1727337670672294, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.673821] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.673824] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670673406, end_time=1727337670673824, runtime=418, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C776-0-0, dag_ret:0, dag_status:2, start_time:1727337670672322, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.673834] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1512, dag_cnt=18, dag_cnts_[dag.get_type()]=18, &dag=0x7f9264c43bd0, dag={ObIDag:{this:0x7f9264c43bd0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C776-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670672322, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8068938909778324349}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:334}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704027, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.673830] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=0] partition merge iter row count(i=0, row_count=80, ghost_row_count=0, pkey={tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4850, key:{tablet_id:{id:333}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223354}, this:0x7f92a3bc4850, timestamp:1727334204223354, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:34, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405529, mt_stat_.create_flush_dag_time:1727337670636870, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.673842] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2880][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C776-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670672319, task_id:YB42C0A8002D-0006230056D8C776-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=334;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.673849] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670673592, end_time=1727337670673848, runtime=256, *this={this:0x7f9287d5e3c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C774-0-0, dag_ret:0, dag_status:2, start_time:1727337670672311, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.673858] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1547, dag_cnt=17, dag_cnts_[dag.get_type()]=17, &dag=0x7f9264c43170, dag={ObIDag:{this:0x7f9264c43170, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C774-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670672311, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:4663792628280492528}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:333}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.673865] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2863][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C774-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670672308, task_id:YB42C0A8002D-0006230056D8C774-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=333;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.673909] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=101066, range=0x7f92aa3d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:5, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:7}, cols_desc:{ObIArray:[column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:7, allocator:0x7f92691aa7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:7, datum_buffer:{capacity:32, datums:0x7f92664f03b0, local_datums:0x7f92664f03b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,]}) [2024-09-26 16:01:10.674038] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.674067] ERROR issue_dba_error (ob_log.cpp:1875) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.674070] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.674079] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=0] partition merge iter row count(i=0, row_count=182, ghost_row_count=0, pkey={tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc5020, key:{tablet_id:{id:101066}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204251158}, this:0x7f92a3bc5020, timestamp:1727334204251158, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:5, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:35, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207971565390001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405546, mt_stat_.create_flush_dag_time:1727337670636885, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.674094] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670673886, end_time=1727337670674094, runtime=208, *this={this:0x7f9287d5e750, type:1, status:2, dag:{ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C775-0-0, dag_ret:0, dag_status:2, start_time:1727337670672316, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.674102] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1786, dag_cnt=16, dag_cnts_[dag.get_type()]=16, &dag=0x7f9264c436a0, dag={ObIDag:{this:0x7f9264c436a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C775-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670672316, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8399557287176188414}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101066}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077692196634026, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.674108] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2875][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C775-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670672314, task_id:YB42C0A8002D-0006230056D8C775-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101066;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.674135] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.674138] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.674144] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=5][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:10.674148] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.674151] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:451}, multi_version_start=1726077691154184001, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.674158] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.674221] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=2] schedule one task(task=0x7f9264afa030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.674228] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=4] succeed to add sys task(task={start_time:1727337670674226, task_id:YB42C0A8002D-0006230056D8C778-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.674232] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=3] schedule one task(task=0x7f9264c449f0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.674238] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=1] succeed to add sys task(task={start_time:1727337670674237, task_id:YB42C0A8002D-0006230056D8C779-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.674240] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=2] schedule one task(task=0x7f9264c44f20, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.674246] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670674245, task_id:YB42C0A8002D-0006230056D8C77A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.674249] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=3] schedule one task(task=0x7f9264c45450, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.674255] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670674254, task_id:YB42C0A8002D-0006230056D8C77B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.674258] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=2] schedule one task(task=0x7f9264c45980, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.674297] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=297, range=0x7f92a9fd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:16, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:18}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b58060, count:6, allocator:0x7f92691aa7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:18, datum_buffer:{capacity:32, datums:0x7f9263b58390, local_datums:0x7f9263b58390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,]}) [2024-09-26 16:01:10.674330] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=17][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.674334] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.674377] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=43][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.674380] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:10.674386] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.674391] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:121}, multi_version_start=1726178585269925002, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.674398] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241548806527000, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.674432] ERROR issue_dba_error (ob_log.cpp:1875) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.674437] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.674448] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=0] partition merge iter row count(i=0, row_count=3, ghost_row_count=0, pkey={tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7eae0, key:{tablet_id:{id:297}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218021}, this:0x7f92a1a7eae0, timestamp:1727334204218021, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:16, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:31, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726207962418314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405020, mt_stat_.ready_for_flush_time:1727334209405497, mt_stat_.create_flush_dag_time:1727337670636836, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.674487] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670674244, end_time=1727337670674487, runtime=243, *this={this:0x7f9264afa030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C772-0-0, dag_ret:0, dag_status:2, start_time:1727337670672294, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.674499] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2204, dag_cnt=15, dag_cnts_[dag.get_type()]=15, &dag=0x7f9264c42710, dag={ObIDag:{this:0x7f9264c42710, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C772-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670672294, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7175089551615529855}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:297}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.674507] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2879][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C772-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670672289, task_id:YB42C0A8002D-0006230056D8C772-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=297;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.674541] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.674545] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.674547] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:10.674612] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=62] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.674618] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=4] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:378}, multi_version_start=1726121601081682000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.674626] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.674678] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.674683] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=5][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.674685] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:10.674688] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.674691] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:215}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.674696] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.674720] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670674717, task_id:YB42C0A8002D-0006230056D8C77C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.674725] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=5] schedule one task(task=0x7f9264c45eb0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.674749] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.674752] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.674754] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:10.674757] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.674759] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=1] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:456}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.674763] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.674788] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337670636922, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.674800] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=14, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.674830] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=25] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.674833] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.674840] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:451}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:5, rowkey_column_count:4, schema_rowkey_col_cnt:2, full_stored_col_cnt:16, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.674858] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:451}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f9263b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:14, store_column_cnt:14, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:2, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:451}, table_count:1, [{i:0, table_key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8650e00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670674130, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=707us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.674895] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C777-0-0, dag_ret:0, dag_status:2, start_time:1727337670673403, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263b1f470, output_block_cnt_arr:0x7f9263b1f478, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337670674130, estimated_finish_time:1727337687674894}) [2024-09-26 16:01:10.674902] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c444c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C777-0-0, dag_ret:0, dag_status:2, start_time:1727337670673403, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.674899] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337670636995, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.674908] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2852][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=6] task finish process(ret=0, start_time=1727337670674129, end_time=1727337670674908, runtime=779, *this={this:0x7f9264c444c0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C777-0-0, dag_ret:0, dag_status:2, start_time:1727337670673403, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.674911] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=11] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753355744, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=10, max_schema_version_in_memtable=1710828753355744, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.674925] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.674942] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=3] schedule one task(task=0x7f926585e030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.674980] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=54] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.674987] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:456}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:12, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753355744, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.675007] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=16] succeed to build merge ctx(tablet_id={id:456}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753355744, storage_schema:{this:0x7f9263a43330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753355744, column_cnt:10, store_column_cnt:10, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:456}, table_count:1, [{i:0, table_key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8634c80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670674745, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=240us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.675032] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=18] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77B-0-0, dag_ret:0, dag_status:2, start_time:1727337670674257, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9263a43490, output_block_cnt_arr:0x7f9263a43498, concurrent_cnt:1, estimate_row_cnt:15, estimate_occupy_size:0, latest_update_ts:1727337670674745, estimated_finish_time:1727337687675032}) [2024-09-26 16:01:10.675039] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c45980, type:15, status:2, dag:{ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77B-0-0, dag_ret:0, dag_status:2, start_time:1727337670674257, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.675045] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2888][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=6] task finish process(ret=0, start_time=1727337670674744, end_time=1727337670675045, runtime=301, *this={this:0x7f9264c45980, type:15, status:2, dag:{ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77B-0-0, dag_ret:0, dag_status:2, start_time:1727337670674257, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.675054] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=4] schedule one task(task=0x7f9264d76030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.675112] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=451, range=0x7f92a9dd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:14, schema_rowkey_cnt:2, rowkey_cnt:4, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:2, column_cnt:16}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d34060, count:4, allocator:0x7f92691aa7d8, capacity:4, init_cnt:4}, datum_utils:{is_oracle_mode:false, rowkey_cnt:4, col_cnt:4, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:16, datum_buffer:{capacity:32, datums:0x7f9287d34350, local_datums:0x7f9287d34350}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,]}) [2024-09-26 16:01:10.675229] ERROR issue_dba_error (ob_log.cpp:1875) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.675235] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=6][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.675245] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc4080, key:{tablet_id:{id:451}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204223009}, this:0x7f92a3bc4080, timestamp:1727334204223009, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:14, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:33, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208018712107000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405021, mt_stat_.ready_for_flush_time:1727334209405579, mt_stat_.create_flush_dag_time:1727337670636922, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.675266] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670675080, end_time=1727337670675266, runtime=186, *this={this:0x7f926585e030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C777-0-0, dag_ret:0, dag_status:2, start_time:1727337670673403, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.675274] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.675275] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1872, dag_cnt=14, dag_cnts_[dag.get_type()]=14, &dag=0x7f9264c44100, dag={ObIDag:{this:0x7f9264c44100, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C777-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670673403, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:7665372330263152188}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:451}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077691154184001, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.675281] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=6][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.675283] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:10.675283] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2883][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C777-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670673398, task_id:YB42C0A8002D-0006230056D8C777-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=451;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.675288] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.675292] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:154}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.675299] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.675315] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670675313, task_id:YB42C0A8002D-0006230056D8C77D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.675319] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=3] schedule one task(task=0x7f9264c463e0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.675332] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752511928, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=0, max_schema_version_in_memtable=0, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.675349] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=14] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.675353] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.675358] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:121}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752511928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.675378] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:121}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670674329, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.03ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.675403] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C779-0-0, dag_ret:0, dag_status:2, start_time:1727337670674240, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f9268515480, output_block_cnt_arr:0x7f9268515488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337670674329, estimated_finish_time:1727337687675403}) [2024-09-26 16:01:10.675411] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.675410] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=7] succeed to init merge ctx(task={this:0x7f9264c44f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C779-0-0, dag_ret:0, dag_status:2, start_time:1727337670674240, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:10.675415] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=4][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.675417] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:10.675416] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2887][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=6] task finish process(ret=0, start_time=1727337670674326, end_time=1727337670675416, runtime=1090, *this={this:0x7f9264c44f20, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C779-0-0, dag_ret:0, dag_status:2, start_time:1727337670674240, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:10.675422] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.675425] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=1] schedule one task(task=0x7f92665d6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.675425] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101015}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.675432] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=4] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.675467] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=1] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337670636973, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.675477] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752725472, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752725472, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.675496] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=15] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.675499] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.675505] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:215}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:4, rowkey_column_count:3, schema_rowkey_col_cnt:1, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752725472, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.675528] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=18] succeed to build merge ctx(tablet_id={id:215}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752725472, storage_schema:{this:0x7f9264b21330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752725472, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:1, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:215}, table_count:1, [{i:0, table_key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b861fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670674673, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=829us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.675557] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77A-0-0, dag_ret:0, dag_status:2, start_time:1727337670674249, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b21460, output_block_cnt_arr:0x7f9264b21468, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337670674673, estimated_finish_time:1727337687675556}) [2024-09-26 16:01:10.675563] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c45450, type:15, status:2, dag:{ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77A-0-0, dag_ret:0, dag_status:2, start_time:1727337670674249, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.675568] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2853][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=5] task finish process(ret=0, start_time=1727337670674672, end_time=1727337670675568, runtime=896, *this={this:0x7f9264c45450, type:15, status:2, dag:{ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77A-0-0, dag_ret:0, dag_status:2, start_time:1727337670674249, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.675577] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=2] schedule one task(task=0x7f9265a04030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.675575] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=121, range=0x7f92aa256200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664f0060, count:5, allocator:0x7f926363e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f92664f0360, local_datums:0x7f92664f0360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:01:10.675604] INFO [STORAGE] dump_info (ob_sstable_struct.cpp:262) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=24] dump merge info(msg="macro block builder close", output_row_per_s=0, new_macro_KB_per_s=0, *this={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:54, merge_start_time:1727337670675549, merge_finish_time:1727337670675603, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:0, max:0, sum:0, count:0}}, {type:"cost_time", info:{min:0, max:0, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:0, max:0, sum:0, count:0}}, {type:"incremental_row_count", info:{min:0, max:0, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:0, snapshot_version:0, start_scn:0, end_scn:0}, macro_id_list:"", comment:""}) [2024-09-26 16:01:10.675628] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=23] partition merge iter row count(i=0, row_count=0, ghost_row_count=0, pkey={tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, table={ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337670636956, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.675634] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=215, range=0x7f92aaad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:1, rowkey_cnt:3, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:1, column_cnt:6}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264d5c060, count:3, allocator:0x7f92a339e7d8, capacity:3, init_cnt:3}, datum_utils:{is_oracle_mode:false, rowkey_cnt:3, col_cnt:3, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9264d5c320, local_datums:0x7f9264d5c320}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:01:10.675642] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1544) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=11] merge macro blocks ok(idx_=0, task={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C779-0-0, dag_ret:0, dag_status:2, start_time:1727337670674240, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:10.675656] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2872][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=9] task finish process(ret=0, start_time=1727337670675521, end_time=1727337670675656, runtime=135, *this={this:0x7f92665d6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C779-0-0, dag_ret:0, dag_status:2, start_time:1727337670674240, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:10.675658] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=456, range=0x7f92a9d52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:10, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:12}, cols_desc:{ObIArray:[column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92729f2060, count:7, allocator:0x7f92b33fe7d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:12, datum_buffer:{capacity:32, datums:0x7f92729f23b0, local_datums:0x7f92729f23b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,]}) [2024-09-26 16:01:10.675682] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337670637026, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.675693] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=10] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92728f9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=18, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.675704] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.675706] ERROR issue_dba_error (ob_log.cpp:1875) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.675707] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.675710] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.675711] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:154}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:20, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.675718] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=0] partition merge iter row count(i=0, row_count=9, ghost_row_count=0, pkey={tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc86d0, key:{tablet_id:{id:215}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204792548}, this:0x7f92a3bc86d0, timestamp:1727334204792548, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752725472, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:42, clock:576716800}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213001362568000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405626, mt_stat_.create_flush_dag_time:1727337670636973, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.675728] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=13] succeed to build merge ctx(tablet_id={id:154}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92728f9330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:18, store_column_cnt:18, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:154}, table_count:1, [{i:0, table_key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8623ce0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670675268, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=442us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.675738] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670675588, end_time=1727337670675738, runtime=150, *this={this:0x7f9265a04030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77A-0-0, dag_ret:0, dag_status:2, start_time:1727337670674249, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.675747] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1498, dag_cnt=13, dag_cnts_[dag.get_type()]=13, &dag=0x7f9264c45090, dag={ObIDag:{this:0x7f9264c45090, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77A-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670674249, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:507863347806783454}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:215}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.675750] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77C-0-0, dag_ret:0, dag_status:2, start_time:1727337670674725, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f9480, output_block_cnt_arr:0x7f92728f9488, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337670675268, estimated_finish_time:1727337687675750}) [2024-09-26 16:01:10.675754] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2861][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77A-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670674245, task_id:YB42C0A8002D-0006230056D8C77A-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=215;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.675755] ERROR issue_dba_error (ob_log.cpp:1875) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.675755] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c45eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77C-0-0, dag_ret:0, dag_status:2, start_time:1727337670674725, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.675760] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2858][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=5] task finish process(ret=0, start_time=1727337670675265, end_time=1727337670675760, runtime=495, *this={this:0x7f9264c45eb0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77C-0-0, dag_ret:0, dag_status:2, start_time:1727337670674725, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.675758] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.675772] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=3] schedule one task(task=0x7f92665d6250, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.675767] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=0] partition merge iter row count(i=0, row_count=45, ghost_row_count=0, pkey={tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6790, key:{tablet_id:{id:456}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204770421}, this:0x7f92a3bc6790, timestamp:1727334204770421, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753355744, max_column_cnt:10, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:38, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213002598666000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405642, mt_stat_.create_flush_dag_time:1727337670636995, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.675778] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=3] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.675782] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670675781, task_id:YB42C0A8002D-0006230056D8C77E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.675784] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=2] schedule one task(task=0x7f9264c46910, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.675800] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=4][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.675797] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670675595, end_time=1727337670675797, runtime=202, *this={this:0x7f9264d76030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77B-0-0, dag_ret:0, dag_status:2, start_time:1727337670674257, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.675803] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.675805] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:10.675809] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.675806] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1549, dag_cnt=12, dag_cnts_[dag.get_type()]=12, &dag=0x7f9264c455c0, dag={ObIDag:{this:0x7f9264c455c0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77B-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670674257, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1604282355669434429}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:456}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.675813] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2882][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77B-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670674254, task_id:YB42C0A8002D-0006230056D8C77B-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=456;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.675812] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=2] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726211867106157000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b863eec0, is_inited:true, wash_score:1727337670636240000, hold_ref_cnt:true, ref_cnt:2, version:2, length:499, tablet_addr:[816017-1849-0-2305843639502065664-2](offset=1921024,size=587,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101016}, data_tablet_id:{id:154}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726074677203981000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726211867106157000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:398898191, row_count:578}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b863f5c0, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:1, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844419038609408-0](offset=1900544,size=1313,type=2,seq=0), type:"MAJOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726074015354550000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305844024975361313-0](offset=1901857,size=946,type=2,seq=0), type:"MINOR", tablet_id:{id:101016}, scn_range:{start_scn:{val:1726074677203981000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726205802467242000, upper_trans_version:1726205802467242000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337670637065, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bd5fc0, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209508808}, this:0x7f92a3bd5fc0, timestamp:1727334209508808, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:69, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277803107894000, v:0}, rec_scn:{val:1726241804194470000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843366771625984-2](offset=1904640,size=333,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843183161774413-0](offset=1904973,size=162,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881251311-0](offset=1905135,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc7f00, 0x7f92a3bd5fc0, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:10.675864] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=52] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101016}, multi_version_start=1726211867106157000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.675871] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.675874] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=154, range=0x7f92a9ed4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:18, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:20}, cols_desc:{ObIArray:[column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9264a62060, count:5, allocator:0x7f92c4a5e7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:20, datum_buffer:{capacity:32, datums:0x7f9264a62360, local_datums:0x7f9264a62360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,]}) [2024-09-26 16:01:10.675899] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670675898, task_id:YB42C0A8002D-0006230056D8C77F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.675903] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=3] schedule one task(task=0x7f9264c46e40, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.675919] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.675921] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.675923] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:10.675926] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.675928] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101063}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.675932] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.675949] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.675980] ERROR issue_dba_error (ob_log.cpp:1875) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.675983] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.675990] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc6f60, key:{tablet_id:{id:154}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204786157}, this:0x7f92a3bc6f60, timestamp:1727334204786157, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:39, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405659, mt_stat_.create_flush_dag_time:1727337670637026, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.676008] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670675837, end_time=1727337670676008, runtime=171, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77C-0-0, dag_ret:0, dag_status:2, start_time:1727337670674725, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676016] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1291, dag_cnt=11, dag_cnts_[dag.get_type()]=11, &dag=0x7f9264c45af0, dag={ObIDag:{this:0x7f9264c45af0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77C-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670674725, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:3553649416163308074}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:154}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.676022] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2877][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77C-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337670674717, task_id:YB42C0A8002D-0006230056D8C77C-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=154;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.676045] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670676043, task_id:YB42C0A8002D-0006230056D8C780-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.676048] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=3] schedule one task(task=0x7f9264c47370, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.676059] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337670637045, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.676071] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.676074] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.676068] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.676075] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=1][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:10.676078] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.676079] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.676081] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:452}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.676081] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.676085] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.676086] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101015}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.676109] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337670637065, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.676118] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752609344, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=4, max_schema_version_in_memtable=1710828752609344, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.676116] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=25] succeed to build merge ctx(tablet_id={id:101015}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92728fb330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101015}, table_count:1, [{i:0, table_key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863a0a0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670675407, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=677us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.676143] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=21] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.676146] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.676147] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=24] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77D-0-0, dag_ret:0, dag_status:2, start_time:1727337670675318, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728fb490, output_block_cnt_arr:0x7f92728fb498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337670675407, estimated_finish_time:1727337687676147}) [2024-09-26 16:01:10.676153] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c463e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77D-0-0, dag_ret:0, dag_status:2, start_time:1727337670675318, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676158] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2867][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=5] task finish process(ret=0, start_time=1727337670675405, end_time=1727337670676158, runtime=753, *this={this:0x7f9264c463e0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77D-0-0, dag_ret:0, dag_status:2, start_time:1727337670675318, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676151] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101016}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:6, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752609344, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.676195] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=32] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337670636939, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.676204] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753116584, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=8, max_schema_version_in_memtable=1710828753116584, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.676185] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=28] succeed to build merge ctx(tablet_id={id:101016}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752609344, storage_schema:{this:0x7f92728f1330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752609344, column_cnt:4, store_column_cnt:4, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:16, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101016}, table_count:1, [{i:0, table_key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b863eec0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670675795, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=354us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.676213] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.676215] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.676219] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=26] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77E-0-0, dag_ret:0, dag_status:2, start_time:1727337670675784, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f1490, output_block_cnt_arr:0x7f92728f1498, concurrent_cnt:1, estimate_row_cnt:32, estimate_occupy_size:0, latest_update_ts:1727337670675795, estimated_finish_time:1727337687676218}) [2024-09-26 16:01:10.676225] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c46910, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77E-0-0, dag_ret:0, dag_status:2, start_time:1727337670675784, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676219] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:378}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:10, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753116584, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.676230] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2864][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=4] task finish process(ret=0, start_time=1727337670675793, end_time=1727337670676229, runtime=436, *this={this:0x7f9264c46910, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77E-0-0, dag_ret:0, dag_status:2, start_time:1727337670675784, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676234] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=11] succeed to build merge ctx(tablet_id={id:378}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753116584, storage_schema:{this:0x7f9264b1f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753116584, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:378}, table_count:1, [{i:0, table_key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864cf80, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670674536, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=1.68ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.676246] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337670637081, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.676252] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C778-0-0, dag_ret:0, dag_status:2, start_time:1727337670674231, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9264b1f480, output_block_cnt_arr:0x7f9264b1f488, concurrent_cnt:1, estimate_row_cnt:2, estimate_occupy_size:0, latest_update_ts:1727337670674536, estimated_finish_time:1727337687676252}) [2024-09-26 16:01:10.676254] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=8] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752913928, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752913928, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.676257] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c449f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C778-0-0, dag_ret:0, dag_status:2, start_time:1727337670674231, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676261] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2886][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=4] task finish process(ret=0, start_time=1727337670674534, end_time=1727337670676261, runtime=1727, *this={this:0x7f9264c449f0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C778-0-0, dag_ret:0, dag_status:2, start_time:1727337670674231, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676264] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=7] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.676267] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.676279] INFO [STORAGE.COMPACTION] get_merged_sstable (ob_tablet_merge_task.cpp:1285) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=5] create new merged sstable(ctx.param_.tablet_id_={id:121}, snapshot_version=1726241132764173999, ctx.param_.merge_type_=3, ctx.create_snapshot_version_=0, table_mode_flag=0) [2024-09-26 16:01:10.676271] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101063}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752913928, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.676287] INFO [STORAGE] close (ob_index_block_builder.cpp:815) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=4] succeed to close sstable index builder(res={root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_root_desc:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), buf:null, height:0, is_meta_root:false}, data_block_ids_.count():0, other_block_ids_.count():0, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, data_column_cnt:10, data_column_checksums:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], row_count:0, max_merged_trans_version:0, contain_uncommitted_row:false, occupy_size:0, original_size:0, data_checksum:0, use_old_macro_block_count:0, compressor_type:1, root_row_store_type:0, nested_offset:0, nested_size:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}) [2024-09-26 16:01:10.676287] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:101063}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752913928, storage_schema:{this:0x7f92684ad330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752913928, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101063}, table_count:1, [{i:0, table_key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b864fe60, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670675914, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=355us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.676304] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=15] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77F-0-0, dag_ret:0, dag_status:2, start_time:1727337670675902, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad4a0, output_block_cnt_arr:0x7f92684ad4a8, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337670675914, estimated_finish_time:1727337687676304}) [2024-09-26 16:01:10.676309] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c46e40, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77F-0-0, dag_ret:0, dag_status:2, start_time:1727337670675902, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676313] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2890][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=4] task finish process(ret=0, start_time=1727337670675912, end_time=1727337670676313, runtime=401, *this={this:0x7f9264c46e40, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77F-0-0, dag_ret:0, dag_status:2, start_time:1727337670675902, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676300] INFO [STORAGE] init (ob_sstable.cpp:118) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=8] succeeded to init sstable(ret=0, this={ObITable:{this:0x7f9268514128, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9268514128, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:null, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:01:10.676325] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337670637096, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.676332] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=7] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828753331944, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=15, max_schema_version_in_memtable=1710828753331944, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.676340] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.676342] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.676327] INFO [STORAGE.COMPACTION] create_sstable (ob_tablet_merge_ctx.cpp:367) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=26] succeed to merge sstable(param={table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_logic_seq:0, schema_version:1710828752511928, create_snapshot_version:0, progressive_merge_round:0, progressive_merge_step:0, is_ready_for_read:true, table_mode:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, index_type:0, root_block_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), root_block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, root_row_store_type:0, latest_row_store_type:1, data_index_tree_height:0, data_block_macro_meta_addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), data_block_macro_meta:{buf:null, size:0, extra_buf:null, extra_size:0, type:0}, index_blocks_cnt:0, data_blocks_cnt:0, micro_block_cnt:0, use_old_macro_block_count:0, row_count:0, rowkey_column_cnt:5, column_cnt:10, column_checksums:[], data_checksum:0, occupy_size:0, original_size:0, max_merged_trans_version:1726241132764173999, ddl_scn:{val:0, v:0}, contain_uncommitted_row:false, is_meta_root:false, compressor_type:1, encrypt_id:0, master_key_id:0, recycle_version:0, nested_offset:0, nested_size:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000"}, table_key={tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, sstable_merge_info={tenant_id:1, ls_id:{id:1}, tablet_id:{id:121}, compaction_scn:1726241548806527000, merge_type:"MINI_MERGE", merge_cost_time:1274, merge_start_time:1727337670674329, merge_finish_time:1727337670675603, dag_id:Y0-0000000000000000-0-0, occupy_size:0, new_flush_occupy_size:0, original_size:0, compressed_size:0, macro_block_count:0, multiplexed_macro_block_count:0, new_micro_count_in_new_macro:0, multiplexed_micro_count_in_new_macro:0, total_row_count:0, incremental_row_count:0, new_flush_data_rate:0, is_full_merge:false, progressive_merge_round:0, progressive_merge_num:0, concurrent_cnt:1, dag_ret:0, task_id:Y0-0000000000000000-0-0, retry_cnt:0, add_time:0, parallel_merge_info:{{type:"scan_units", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"cost_time", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"use_old_macro_block_cnt", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, {type:"incremental_row_count", info:{min:9223372036854775807, max:-9223372036854775808, sum:0, count:0}}, }, filter_statistics:CompactionFilterStatistics:{{type:"NOT_CHANGE", info:0}, {type:"REMOVE", info:0}, }, participant_table_info:{is_major_merge:false, table_cnt:1, snapshot_version:0, start_scn:1726211867305630000, end_scn:1726241548806527000}, macro_id_list:"", comment:""}) [2024-09-26 16:01:10.676346] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:452}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:17, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828753331944, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.676359] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=10] succeed to build merge ctx(tablet_id={id:452}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828753331944, storage_schema:{this:0x7f92728f3330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828753331944, column_cnt:15, store_column_cnt:15, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:452}, table_count:1, [{i:0, table_key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8652d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670676068, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=277us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.676385] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=22] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C780-0-0, dag_ret:0, dag_status:2, start_time:1727337670676048, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f3480, output_block_cnt_arr:0x7f92728f3488, concurrent_cnt:1, estimate_row_cnt:10, estimate_occupy_size:0, latest_update_ts:1727337670676068, estimated_finish_time:1727337687676385}) [2024-09-26 16:01:10.676390] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c47370, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C780-0-0, dag_ret:0, dag_status:2, start_time:1727337670676048, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676395] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2870][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=5] task finish process(ret=0, start_time=1727337670676066, end_time=1727337670676395, runtime=329, *this={this:0x7f9264c47370, type:15, status:2, dag:{ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C780-0-0, dag_ret:0, dag_status:2, start_time:1727337670676048, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676422] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=2] schedule one task(task=0x7f9287c66030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=2, running_task_cnts_[priority]=2, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.676428] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=3] schedule one task(task=0x7f9264c80030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.676431] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=2] schedule one task(task=0x7f927282a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.676433] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=2] schedule one task(task=0x7f9264c48ed0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.676436] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=2] schedule one task(task=0x7f9287c663c0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.676491] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101063, range=0x7f92aa352200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d38060, count:8, allocator:0x7f92b32407d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f9287d383e0, local_datums:0x7f9287d383e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:10.676566] ERROR issue_dba_error (ob_log.cpp:1875) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.676570] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.676576] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101016, range=0x7f92aa7d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287dd2060, count:6, allocator:0x7f92691ba7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9287dd2390, local_datums:0x7f9287dd2390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:01:10.676580] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=0] partition merge iter row count(i=0, row_count=1, ghost_row_count=0, pkey={tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bca610, key:{tablet_id:{id:101063}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046710}, this:0x7f92a3bca610, timestamp:1727334208046710, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752913928, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:46, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236008922704000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405709, mt_stat_.create_flush_dag_time:1727337670637081, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.676615] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670676447, end_time=1727337670676615, runtime=168, *this={this:0x7f9264c48ed0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77F-0-0, dag_ret:0, dag_status:2, start_time:1727337670675902, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676625] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=723, dag_cnt=10, dag_cnts_[dag.get_type()]=10, &dag=0x7f9264c46a80, dag={ObIDag:{this:0x7f9264c46a80, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77F-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670675902, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8720304653345806796}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101063}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.676633] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2874][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77F-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670675898, task_id:YB42C0A8002D-0006230056D8C77F-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101063;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.676663] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670676662, task_id:YB42C0A8002D-0006230056D8C781-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.676667] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=3] schedule one task(task=0x7f9264c478a0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.676691] ERROR issue_dba_error (ob_log.cpp:1875) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.676695] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.676704] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7f00, key:{tablet_id:{id:101016}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204789577}, this:0x7f92a3bc7f00, timestamp:1727334204789577, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:41, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405692, mt_stat_.create_flush_dag_time:1727337670637065, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.676726] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670676522, end_time=1727337670676726, runtime=204, *this={this:0x7f927282a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77E-0-0, dag_ret:0, dag_status:2, start_time:1727337670675784, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676737] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=952, dag_cnt=9, dag_cnts_[dag.get_type()]=9, &dag=0x7f9264c46550, dag={ObIDag:{this:0x7f9264c46550, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77E-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670675784, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-3976350754940852513}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101016}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.676744] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2871][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77E-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670675781, task_id:YB42C0A8002D-0006230056D8C77E-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101016;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.676757] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670676756, task_id:YB42C0A8002D-0006230056D8C782-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.676760] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=3] schedule one task(task=0x7f9264c47dd0, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.676777] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.676778] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=35][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.676780] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.676782] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.676782] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:10.676786] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.676787] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=4][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:10.676789] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:330}, multi_version_start=1726077706539141067, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.676791] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=2] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.676795] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.676795] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:328}, multi_version_start=1726077706139416032, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.676801] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.676846] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=452, range=0x7f92a9ad4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:15, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:17}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5a060, count:5, allocator:0x7f9275d787d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:17, datum_buffer:{capacity:32, datums:0x7f9263b5a360, local_datums:0x7f9263b5a360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,]}) [2024-09-26 16:01:10.676908] ERROR issue_dba_error (ob_log.cpp:1875) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.676911] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.676917] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=0] partition merge iter row count(i=0, row_count=10, ghost_row_count=0, pkey={tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9e40, key:{tablet_id:{id:452}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208046081}, this:0x7f92a3bc9e40, timestamp:1727334208046081, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753331944, max_column_cnt:15, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:45, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236156720685000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405727, mt_stat_.create_flush_dag_time:1727337670637096, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.676934] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670676817, end_time=1727337670676934, runtime=117, *this={this:0x7f9287c663c0, type:1, status:2, dag:{ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C780-0-0, dag_ret:0, dag_status:2, start_time:1727337670676048, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.676942] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=894, dag_cnt=8, dag_cnts_[dag.get_type()]=8, &dag=0x7f9264c46fb0, dag={ObIDag:{this:0x7f9264c46fb0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C780-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670676048, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1830562711513853292}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:452}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.676949] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2885][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C780-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670676043, task_id:YB42C0A8002D-0006230056D8C780-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=452;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.676961] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670676959, task_id:YB42C0A8002D-0006230056D8C783-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.676965] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=3] schedule one task(task=0x7f9264c48300, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.676980] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=5][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.676983] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=3][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.676985] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726149987022722000) [2024-09-26 16:01:10.676989] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.676991] INFO [STORAGE] build_new_table_store (ob_tablet_table_store.cpp:1206) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=19] succeed to build new table store(major_tables_={cnt:1, is_inited:true}, minor_tables_={cnt:3, is_inited:true}, memtables_={count:1}, PRINT_TS(*this)={ObTabletTableStore_Pretty:this:0x7f92ab151ca8, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, memtables:{count:1}, is_ready_for_read:true, table_array:{ [2850] [ YB42C0A8002D-0006230056D8C779-0-0 ] table_arr table_type upper_trans_ver max_merge_ver start_scn end_scn ref uncommit_row [2850] [ YB42C0A8002D-0006230056D8C779-0-0 ] MAJOR MAJOR 1725120003420849000 1725120003420849000 {val:0, v:0} {val:1726077600415496000, v:0} 0 false [2850] [ YB42C0A8002D-0006230056D8C779-0-0 ] MINOR MINOR 1726178585269925002 1726178585269925002 {val:1726077259159067000, v:0} {val:1726178585269925002, v:0} 0 false [2850] [ YB42C0A8002D-0006230056D8C779-0-0 ] MINI 1726211867106157000 1726211867106157000 {val:1726178585269925002, v:0} {val:1726211867305630000, v:0} 0 false [2850] [ YB42C0A8002D-0006230056D8C779-0-0 ] MINI 1726241132764173999 1726241132764173999 {val:1726211867305630000, v:0} {val:1726241548806527000, v:0} 0 false [2850] [ YB42C0A8002D-0006230056D8C779-0-0 ] MEM ACTIVE 9223372036854775807 18446744073709547609 {val:1726241548806527000, v:0} {val:4611686018427387903, v:0} 3 unused }}) [2024-09-26 16:01:10.676991] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=2] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:329}, multi_version_start=1726077706339704000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.677008] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=14] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.677033] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337670637130, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.677044] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=10] update column cnt(column_cnt_=12, store_column_cnt_=12, column_cnt_=12, column_array_.count()=0) [2024-09-26 16:01:10.677041] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=5] multi version scan iterator init succ(param.table_id_=378, range=0x7f92a9e56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:8, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:10}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263a3e060, count:5, allocator:0x7f92636387d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:10, datum_buffer:{capacity:32, datums:0x7f9263a3e360, local_datums:0x7f9263a3e360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,]}) [2024-09-26 16:01:10.677047] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=3] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92728f7490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=12, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.677061] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=12][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.677067] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=10] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.677070] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.677076] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92728f7490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:330}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:8, rowkey_column_count:7, schema_rowkey_col_cnt:5, full_stored_col_cnt:14, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=24 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.677090] ERROR issue_dba_error (ob_log.cpp:1875) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.677092] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.677100] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=0] partition merge iter row count(i=0, row_count=2, ghost_row_count=0, pkey={tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a1a7f2b0, key:{tablet_id:{id:378}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204218231}, this:0x7f92a1a7f2b0, timestamp:1727334204218231, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828753116584, max_column_cnt:8, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:32, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726208020155007000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209405596, mt_stat_.create_flush_dag_time:1727337670636939, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.677097] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:330}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92728f7490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:12, store_column_cnt:12, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:5, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:22, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:330}, table_count:1, [{i:0, table_key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e107fe0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670676772, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=301us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.677138] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670677018, end_time=1727337670677138, runtime=120, *this={this:0x7f9287c66030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C778-0-0, dag_ret:0, dag_status:2, start_time:1727337670674231, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.677141] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=31] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C782-0-0, dag_ret:0, dag_status:2, start_time:1727337670676760, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92728f75f0, output_block_cnt_arr:0x7f92728f75f8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337670676772, estimated_finish_time:1727337687677140}) [2024-09-26 16:01:10.677147] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c47dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C782-0-0, dag_ret:0, dag_status:2, start_time:1727337670676760, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.677148] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=2917, dag_cnt=7, dag_cnts_[dag.get_type()]=7, &dag=0x7f9264c44630, dag={ObIDag:{this:0x7f9264c44630, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C778-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670674231, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6759548225246904317}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:378}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726121601081682000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.677152] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2849][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=5] task finish process(ret=0, start_time=1727337670676770, end_time=1727337670677152, runtime=382, *this={this:0x7f9264c47dd0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C782-0-0, dag_ret:0, dag_status:2, start_time:1727337670676760, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.677155] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2876][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C778-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670674226, task_id:YB42C0A8002D-0006230056D8C778-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=378;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.677181] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337670637113, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.677190] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=9] update column cnt(column_cnt_=26, store_column_cnt_=26, column_cnt_=26, column_array_.count()=0) [2024-09-26 16:01:10.677194] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=2] succeed to add sys task(task={start_time:1727337670677192, task_id:YB42C0A8002D-0006230056D8C784-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.677197] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=3] schedule one task(task=0x7f9264c48830, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.677195] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=5] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f9287db5480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=26, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.677208] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=9] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.677211] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.677213] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=6][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.677215] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=2][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.677218] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=3][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:10.677216] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f9287db5480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:328}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:6, rowkey_column_count:5, schema_rowkey_col_cnt:3, full_stored_col_cnt:28, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.677233] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.677237] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=3] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:332}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.677243] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=3] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.677242] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=14] succeed to build merge ctx(tablet_id={id:328}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f9287db5480, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:26, store_column_cnt:26, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:328}, table_count:1, [{i:0, table_key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e072bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670676774, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=440us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.677268] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=3] schedule one task(task=0x7f9264d76030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.677266] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C781-0-0, dag_ret:0, dag_status:2, start_time:1727337670676666, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9287db55d0, output_block_cnt_arr:0x7f9287db55d8, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337670676774, estimated_finish_time:1727337687677266}) [2024-09-26 16:01:10.677272] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c478a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C781-0-0, dag_ret:0, dag_status:2, start_time:1727337670676666, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.677277] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2862][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=5] task finish process(ret=0, start_time=1727337670676772, end_time=1727337670677277, runtime=505, *this={this:0x7f9264c478a0, type:15, status:2, dag:{ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C781-0-0, dag_ret:0, dag_status:2, start_time:1727337670676666, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.677320] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=6] multi version scan iterator init succ(param.table_id_=330, range=0x7f92aac56200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:12, schema_rowkey_cnt:5, rowkey_cnt:7, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:5, column_cnt:14}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=22 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9287d2e060, count:7, allocator:0x7f92636327d8, capacity:7, init_cnt:7}, datum_utils:{is_oracle_mode:false, rowkey_cnt:7, col_cnt:7, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:14, datum_buffer:{capacity:32, datums:0x7f9287d2e3b0, local_datums:0x7f9287d2e3b0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,]}) [2024-09-26 16:01:10.677522] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=14][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.677557] ERROR issue_dba_error (ob_log.cpp:1875) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=4][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.677560] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.677579] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=0] partition merge iter row count(i=0, row_count=141, ghost_row_count=0, pkey={tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc9670, key:{tablet_id:{id:330}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208016328}, this:0x7f92a3bc9670, timestamp:1727334208016328, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:12, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:44, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405761, mt_stat_.create_flush_dag_time:1727337670637130, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.677599] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670677278, end_time=1727337670677599, runtime=321, *this={this:0x7f9264d76030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C782-0-0, dag_ret:0, dag_status:2, start_time:1727337670676760, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.677609] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=849, dag_cnt=6, dag_cnts_[dag.get_type()]=6, &dag=0x7f9264c47a10, dag={ObIDag:{this:0x7f9264c47a10, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C782-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670676760, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-6933628521298738662}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:330}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706539141067, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.677618] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2856][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C782-0-0] [lt=7] succeed to del sys task(removed_task={start_time:1727337670676756, task_id:YB42C0A8002D-0006230056D8C782-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=330;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.677640] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=2] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337670637165, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.677649] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=9] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f92664ef330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=25, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.677660] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=8] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.677663] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=2] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.677667] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=2] success to init data desc as index(ret=0, merge_schema={this:0x7f92664ef330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:332}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:27, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.677686] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=15] succeed to build merge ctx(tablet_id={id:332}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f92664ef330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:25, store_column_cnt:25, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:332}, table_count:1, [{i:0, table_key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8648b00, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670677209, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=457us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.677706] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=16] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C784-0-0, dag_ret:0, dag_status:2, start_time:1727337670677197, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92664ef490, output_block_cnt_arr:0x7f92664ef498, concurrent_cnt:1, estimate_row_cnt:5, estimate_occupy_size:0, latest_update_ts:1727337670677209, estimated_finish_time:1727337687677706}) [2024-09-26 16:01:10.677712] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c48830, type:15, status:2, dag:{ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C784-0-0, dag_ret:0, dag_status:2, start_time:1727337670677197, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.677717] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2848][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=5] task finish process(ret=0, start_time=1727337670677208, end_time=1727337670677717, runtime=509, *this={this:0x7f9264c48830, type:15, status:2, dag:{ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C784-0-0, dag_ret:0, dag_status:2, start_time:1727337670677197, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.677731] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=10] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337670637147, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.677740] INFO [STORAGE] update_column_cnt (ob_storage_schema.cpp:987) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=8] update column cnt(column_cnt_=29, store_column_cnt_=29, column_cnt_=29, column_array_.count()=0) [2024-09-26 16:01:10.677742] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=2] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752994696, storage_schema:{this:0x7f92684ad490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=false, max_column_cnt_in_memtable=29, max_schema_version_in_memtable=1710828752994696, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.677750] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=6] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=3, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.677753] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.677757] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=3] success to init data desc as index(ret=0, merge_schema={this:0x7f92684ad490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:329}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:7, rowkey_column_count:6, schema_rowkey_col_cnt:4, full_stored_col_cnt:31, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752994696, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=23 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.677772] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=12] succeed to build merge ctx(tablet_id={id:329}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752994696, storage_schema:{this:0x7f92684ad490, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752994696, column_cnt:29, store_column_cnt:29, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:4, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:21, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:329}, table_count:1, [{i:0, table_key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f926e089f20, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670676976, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=780us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.677790] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=13] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C783-0-0, dag_ret:0, dag_status:2, start_time:1727337670676965, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f92684ad5f0, output_block_cnt_arr:0x7f92684ad5f8, concurrent_cnt:1, estimate_row_cnt:62, estimate_occupy_size:0, latest_update_ts:1727337670676976, estimated_finish_time:1727337687677790}) [2024-09-26 16:01:10.677795] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=5] succeed to init merge ctx(task={this:0x7f9264c48300, type:15, status:2, dag:{ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C783-0-0, dag_ret:0, dag_status:2, start_time:1727337670676965, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.677800] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2884][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=4] task finish process(ret=0, start_time=1727337670676975, end_time=1727337670677800, runtime=825, *this={this:0x7f9264c48300, type:15, status:2, dag:{ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C783-0-0, dag_ret:0, dag_status:2, start_time:1727337670676965, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.677828] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=3] schedule one task(task=0x7f9265aa6030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=3, running_task_cnts_[priority]=3, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.677833] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=4] schedule one task(task=0x7f9262cbe030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=4, running_task_cnts_[priority]=4, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.677838] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=2] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=5, running_task_cnts_[priority]=5, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.677843] INFO [SERVER] add_task (ob_sys_task_stat.cpp:143) [2891][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=3] succeed to add sys task(task={start_time:1727337670677842, task_id:YB42C0A8002D-0006230056D8C785-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.677847] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=3] schedule one task(task=0x7f9264c48d60, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=6, running_task_cnts_[priority]=6, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.677882] WDIAG [STORAGE] inner_get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:352) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=3][errcode=-4018] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.677890] WDIAG [STORAGE] get_neighbour_major_freeze (ob_tenant_freeze_info_mgr.cpp:388) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=7][errcode=-4018] failed to get neighbour_major_freeze(ret=-4018, ret="OB_ENTRY_NOT_EXIST") [2024-09-26 16:01:10.677893] WDIAG [STORAGE] get_neighbour_freeze_info (ob_partition_merge_policy.cpp:68) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=2][errcode=-4018] Failed to get freeze info, use snapshot_gc_ts instead(ret=-4018, snapshot_version=1726211867106157000) [2024-09-26 16:01:10.677897] INFO [STORAGE] get_freeze_info_behind_snapshot_version_ (ob_tenant_freeze_info_mgr.cpp:315) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=1] no freeze info in curr info_list(ret=-4018, cur_idx_=0, info_list_[0]=[], info_list_[1]=[]) [2024-09-26 16:01:10.677896] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=328, range=0x7f92ab1d0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:26, schema_rowkey_cnt:3, rowkey_cnt:5, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:3, column_cnt:28}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92684e6060, count:5, allocator:0x7f92691ba7d8, capacity:5, init_cnt:5}, datum_utils:{is_oracle_mode:false, rowkey_cnt:5, col_cnt:5, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:28, datum_buffer:{capacity:32, datums:0x7f92684e6360, local_datums:0x7f92684e6360}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,]}) [2024-09-26 16:01:10.677927] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=329, range=0x7f92aa952200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:29, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:31}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665ae060, count:6, allocator:0x7f92b321a7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:31, datum_buffer:{capacity:32, datums:0x7f92665ae390, local_datums:0x7f92665ae390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,col_id=27:NOP,col_id=28:NOP,col_id=29:NOP,col_id=30:NOP,]}) [2024-09-26 16:01:10.677901] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4026) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=3] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, old_min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, old_tablet_multi_version_start=1726149987022722000, old_tablet_snaphsot_version=1726211867106157000, tablet={this:0x7f92b86a2d40, is_inited:true, wash_score:1727337670636240000, hold_ref_cnt:true, ref_cnt:2, version:2, length:511, tablet_addr:[816017-1849-0-2305843652387110912-2](offset=2064384,size=599,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:101067}, data_tablet_id:{id:334}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726064245250507001, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726149987022722000, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:1854554341, row_count:13383}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b86a3488, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844595132416000-0](offset=2048000,size=1477,type=2,seq=0), type:"MAJOR", tablet_id:{id:101067}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1726063562807034000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843951961064901-0](offset=2049477,size=878,type=2,seq=0), type:"MINI", tablet_id:{id:101067}, scn_range:{start_scn:{val:1726064245250507001, v:0}, end_scn:{val:1726149987022722000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726149986147010000, upper_trans_version:1726149986147010000}, {i:1, addr:[816017-1849-0-2305843958403516723-0](offset=2050355,size=884,type=2,seq=0), type:"MINI", tablet_id:{id:101067}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726208018709503000, upper_trans_version:1726208018709503000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:4, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337670637185, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bde490, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241290462324999, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334221276956}, this:0x7f92a3bde490, timestamp:1727334221276956, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:86, clock:639631360}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726265292880560000, v:0}, rec_scn:{val:1726264918599518000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843482735894528-2](offset=2056192,size=441,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843196046827961-0](offset=2056633,size=174,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251881402983-0](offset=2056807,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bcbd80, 0x7f92a3bde490, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}) [2024-09-26 16:01:10.677948] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4032) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=46] get multi version start(ret=0, ls_id={id:1}, tablet_id={id:101067}, multi_version_start=1726149987022722000, min_reserved_snapshot=0, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1726276007200618000, last_major_snapshot_version=1726077600415496000) [2024-09-26 16:01:10.677954] INFO [STORAGE] check_tx_table_ready (ob_tenant_tablet_scheduler.cpp:986) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=2] tx table ready(sstable_end_scn={val:1726241290462324999, v:0}, max_decided_scn={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.677984] ERROR issue_dba_error (ob_log.cpp:1875) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.677987] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.677996] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc8ea0, key:{tablet_id:{id:328}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208005978}, this:0x7f92a3bc8ea0, timestamp:1727334208005978, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:26, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:43, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405744, mt_stat_.create_flush_dag_time:1727337670637113, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.678016] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670677864, end_time=1727337670678015, runtime=151, *this={this:0x7f9265aa6030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C781-0-0, dag_ret:0, dag_status:2, start_time:1727337670676666, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.678024] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1358, dag_cnt=5, dag_cnts_[dag.get_type()]=5, &dag=0x7f9264c474e0, dag={ObIDag:{this:0x7f9264c474e0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C781-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670676666, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-1536604874905918360}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:328}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706139416032, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.678031] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2851][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C781-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670676662, task_id:YB42C0A8002D-0006230056D8C781-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=328;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.678088] ERROR issue_dba_error (ob_log.cpp:1875) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.678093] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=5][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf4eba37 0xf4ff42b 0xf21785b 0xf2132e8 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.678102] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=2] multi version scan iterator init succ(param.table_id_=332, range=0x7f92aacd4200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:25, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:27}, cols_desc:{ObIArray:[column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92665f8060, count:6, allocator:0x7f92c244a7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:27, datum_buffer:{capacity:32, datums:0x7f92665f8390, local_datums:0x7f92665f8390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,col_id=8:NOP,col_id=9:NOP,col_id=10:NOP,col_id=11:NOP,col_id=12:NOP,col_id=13:NOP,col_id=14:NOP,col_id=15:NOP,col_id=16:NOP,col_id=17:NOP,col_id=18:NOP,col_id=19:NOP,col_id=20:NOP,col_id=21:NOP,col_id=22:NOP,col_id=23:NOP,col_id=24:NOP,col_id=25:NOP,col_id=26:NOP,]}) [2024-09-26 16:01:10.678104] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=0] partition merge iter row count(i=0, row_count=47, ghost_row_count=0, pkey={tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcb5b0, key:{tablet_id:{id:329}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208126106}, this:0x7f92a3bcb5b0, timestamp:1727334208126106, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752994696, max_column_cnt:29, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:48, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236164644015001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405777, mt_stat_.create_flush_dag_time:1727337670637147, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.678128] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670677873, end_time=1727337670678128, runtime=255, *this={this:0x7f9262cbe030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C783-0-0, dag_ret:0, dag_status:2, start_time:1727337670676965, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.678139] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1174, dag_cnt=4, dag_cnts_[dag.get_type()]=4, &dag=0x7f9264c47f40, dag={ObIDag:{this:0x7f9264c47f40, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C783-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670676965, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:1056199536334423564}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:329}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726077706339704000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726149987022722000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.678147] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2866][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C783-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670676959, task_id:YB42C0A8002D-0006230056D8C783-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=329;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.678158] ERROR issue_dba_error (ob_log.cpp:1875) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.678161] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.678169] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=0] partition merge iter row count(i=0, row_count=5, ghost_row_count=0, pkey={tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcade0, key:{tablet_id:{id:332}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208081639}, this:0x7f92a3bcade0, timestamp:1727334208081639, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:25, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:47, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236175119148001, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405024, mt_stat_.ready_for_flush_time:1727334209405794, mt_stat_.create_flush_dag_time:1727337670637165, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.678185] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670678065, end_time=1727337670678185, runtime=120, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C784-0-0, dag_ret:0, dag_status:2, start_time:1727337670677197, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.678193] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=996, dag_cnt=3, dag_cnts_[dag.get_type()]=3, &dag=0x7f9264c48470, dag={ObIDag:{this:0x7f9264c48470, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C784-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670677197, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-7405463607918801432}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:332}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.678199] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2857][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C784-0-0] [lt=5] succeed to del sys task(removed_task={start_time:1727337670677192, task_id:YB42C0A8002D-0006230056D8C784-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=332;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.678196] INFO [MDS] init_for_evict_medium_info (ob_tablet_mds_data.cpp:414) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=14] succeeded to init mds data(ret=0, finish_medium_scn=1726077600415496000, src_medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, src_extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list={ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, extra_medium_info={info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}) [2024-09-26 16:01:10.678213] INFO [STORAGE] init_for_merge (ob_tablet.cpp:363) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=7] succeeded to init tablet for mini/minor/major merge(ret=0, param={sstable:0x7f9268514128, snapshot_version:1726241132764173999, clog_checkpoint_scn:{val:1726241548806527000, v:0}, multi_version_start:1726178585269925002, need_report:false, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, rebuild_seq:0, update_with_major_flag:false, need_check_sstable:false, ddl_info:{keep_old_ddl_sstable:true, ddl_start_scn:{val:0, v:0}, ddl_commit_scn:{val:0, v:0}, ddl_checkpoint_scn:{val:0, v:0}, ddl_snapshot_version:0, ddl_execution_id:-1, data_format_version:0}, allow_duplicate_sstable:false, merge_type:"MINI_MERGE", need_check_transfer_seq:true, transfer_seq:0}, old_tablet={this:0x7f92b8626bc0, is_inited:true, wash_score:1727337670636240000, hold_ref_cnt:true, ref_cnt:3, version:2, length:503, tablet_addr:[816017-1849-0-2305843643796406272-2](offset=1294336,size=591,type=2,seq=2), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726211867305630000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726211867106157000, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f92b862729c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:2, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:2}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bc5fc0, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204623036}, this:0x7f92a3bc5fc0, timestamp:1727334204623036, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:37, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241548806527000, v:0}, rec_scn:{val:1726211870220940000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405022, mt_stat_.ready_for_flush_time:1727334209455996, mt_stat_.create_flush_dag_time:1727337670636956, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}, {i:1, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[816017-1849-0-2305843479513903104-2](offset=1290240,size=438,type=2,seq=2)}, storage_schema_addr:{ptr:null, addr:[816017-1849-0-2305843176718709174-0](offset=1290678,size=156,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843251880637010-0](offset=1290834,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[816017-1849-0-2305843216447157044-0](offset=1291060,size=193,type=2,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bc5fc0, 0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:2}, this={this:0x7f9287db2060, is_inited:true, wash_score:-9223372036854775808, hold_ref_cnt:true, ref_cnt:1, version:2, length:0, tablet_addr:[0-0-0-3458765810900664320-0](offset=0,size=1208,type=3,seq=0), allocator:0x7f92ab152450, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:121}, data_tablet_id:{id:121}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1726077259159067000, v:0}, clog_checkpoint_scn:{val:1726241548806527000, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1726241132764173999, multi_version_start:1726178585269925002, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1726077600415496000, cur_report_version:1726077600415496000, data_checksum:3587602083, row_count:14}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1726083780823179000, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0}, table_store_addr:{ptr:{this:0x7f9287db2670, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:3, is_inited:true}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:1}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[816017-1849-0-2305844332064907264-0](offset=1286144,size=1232,type=2,seq=0), type:"MAJOR", tablet_id:{id:121}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}, snapshot_version:1726077600415496000, max_merge_version:1725120003420849000}}, minor_tables:{{i:0, addr:[816017-1849-0-2305843335632495824-0](offset=1287376,size=304,type=2,seq=0), type:"MINOR", tablet_id:{id:121}, scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726178585269925002, upper_trans_version:1726178585269925002}, {i:1, addr:[816017-1849-0-2305843327042561536-0](offset=1287680,size=296,type=2,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726211867106157000, upper_trans_version:1726211867106157000}, {i:2, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), type:"MINI", tablet_id:{id:121}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, contain_uncommitted_row:"no", max_merge_version:1726241132764173999, upper_trans_version:1726241132764173999}}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{{i:0, memtable:{ObITable:{this:0x7f92a3bccd20, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726241548806527000, v:0}, end_scn:{val:4611686018427387903, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334209405066}, this:0x7f92a3bccd20, timestamp:1727334209405066, state:0, freeze_clock:2, max_schema_version:0, max_data_schema_version:0, max_column_cnt:0, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:1, id:50, clock:610271232}, host:0x7f92ba380030, arena_handle:{allocated:2097152}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726277810029040000, v:0}, rec_scn:{val:1726241553809444000, v:0}, snapshot_version:{val:4611686018427387903, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:0, allow_freeze:true, mt_stat_.frozen_time:0, mt_stat_.ready_for_flush_time:0, mt_stat_.create_flush_dag_time:0, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}}}}}, addr:[0-0-0-3458764857417924608-0](offset=0,size=320,type=3,seq=0)}, storage_schema_addr:{ptr:{this:0x7f9287db2520, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}, addr:[0-0-0-3458764814468251648-0](offset=0,size=280,type=3,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9287db3880, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:0x7f9287db3950, dump_node:{status:SET|TRANS|ON_COMMIT|False, user_data:{redefined:false, snapshot_version:0, schema_version:0, data_tablet_id:{id:0}, hidden_tablet_id:{id:0}, lob_meta_tablet_id:{id:50121}, lob_piece_tablet_id:{id:60121}, is_old_mds:true}}, addr:[0-0-0-3458764634079625216-0](offset=0,size=112,type=3,seq=0)}}, extra_medium_info:{info:257, compat:1, last_compaction_type:1, wait_check_flag:0, last_medium_scn:1726077600415496000}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1710828752147978000, v:0}, create_commit_version:1710828752147978000, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[0x7f92a3bccd20, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:1}) [2024-09-26 16:01:10.678323] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=95] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9264b1e060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MAJOR", scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1726077600415496000, v:0}}}, ref_cnt:0, upper_trans_version:1725120003420849000, timestamp:0}, this:0x7f9264b1e060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1725120003420849000, max_merged_trans_version:1725120003420849000, data_macro_block_count:1, nested_size:4096, nested_offset:589824, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:84, row_count_:14, occupy_size_:1604, original_size_:1914, data_checksum_:3587602083, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:1, data_micro_block_count_:1, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:2, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1725120003420849000, max_merged_trans_version_:1725120003420849000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:2, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9264b1e310, column_checksum_count:10, data_root_info:{addr:[0-0-0-3458764933653594112-0](offset=0,size=391,type=3,seq=0), block_data:{buf:0x7f9264b1e360, size:391, extra_buf:0x7f9264b1e4e7, extra_size:424, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-3458765009889263616-0](offset=0,size=462,type=3,seq=0), block_data:{buf:0x7f9264b1e68f, size:462, extra_buf:null, extra_size:0, type:1}}, data_block_count:1, other_block_count:0, linked_block_count:0, data_block_ids:0x7f9264b1e85d, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:true, nested_offset:589824, nested_size:4096}}, valid_for_reading:true}) [2024-09-26 16:01:10.678343] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=17] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9264b22060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1726077259159067000, v:0}, end_scn:{val:1726178585269925002, v:0}}}, ref_cnt:0, upper_trans_version:1726178585269925002, timestamp:0}, this:0x7f9264b22060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726178585269925002, max_merged_trans_version:1726178585269925002, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726178585269925002, v:0}, meta:{basic_meta:{version:1, length:94, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726178585269925002, max_merged_trans_version_:1726178585269925002, recycle_version:1726077600415496000, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726178585269925002, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:1, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9264b22310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:01:10.678337] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=4] multi version scan iterator init succ(param.table_id_=101015, range=0x7f92aafd0200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:4, schema_rowkey_cnt:4, rowkey_cnt:6, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:4, column_cnt:6}, cols_desc:{ObIArray:[column_id=19 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=16 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f9263b5c060, count:6, allocator:0x7f92691ba7d8, capacity:6, init_cnt:6}, datum_utils:{is_oracle_mode:false, rowkey_cnt:6, col_cnt:6, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:6, datum_buffer:{capacity:32, datums:0x7f9263b5c390, local_datums:0x7f9263b5c390}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,]}) [2024-09-26 16:01:10.678354] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=10] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9264b1e060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726178585269925002, v:0}, end_scn:{val:1726211867305630000, v:0}}}, ref_cnt:0, upper_trans_version:1726211867106157000, timestamp:0}, this:0x7f9264b1e060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726211867106157000, max_merged_trans_version:1726211867106157000, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726211867305630000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726211867106157000, max_merged_trans_version_:1726211867106157000, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726211867305630000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9264b1e310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:01:10.678364] INFO [STORAGE] serialize (ob_sstable.cpp:896) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=9] succeed to serialize sstable(status.pack_=21762, this={ObITable:{this:0x7f9264b22060, key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref_cnt:0, upper_trans_version:1726241132764173999, timestamp:0}, this:0x7f9264b22060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:1726241132764173999, max_merged_trans_version:1726241132764173999, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1726241548806527000, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:5, column_cnt_:10, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1710828752511928, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1726241132764173999, max_merged_trans_version_:1726241132764173999, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1726241548806527000, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x7f9264b22310, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}) [2024-09-26 16:01:10.678374] WDIAG [SHARE] alloc_block (ob_local_device.cpp:719) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=9][errcode=-4184] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:10.678418] ERROR issue_dba_error (ob_log.cpp:1875) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=3][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.678421] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=3][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf9de5b1 0xf9dcfd6 0xe51c6fb 0xe519c03 0xe518f89 0xe516f8c 0xf01158d 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.678452] ERROR issue_dba_error (ob_log.cpp:1875) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.678455] INFO [STORAGE] reset (ob_tablet.cpp:159) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=0] reset tablet(this=0x7f9287db2060, ls_id={id:1}, tablet_id={id:121}, lbt()="0x121f1f9c 0xe52cc09 0xe52c6c5 0xe52bd13 0x4fd348f 0xf01110c 0xf25bd53 0xf25b332 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:10.678456] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=4][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.678467] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=0] partition merge iter row count(i=0, row_count=32, ghost_row_count=0, pkey={tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bc7730, key:{tablet_id:{id:101015}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334204787799}, this:0x7f92a3bc7730, timestamp:1727334204787799, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752609344, max_column_cnt:4, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:40, clock:574619648}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726213007698282000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405023, mt_stat_.ready_for_flush_time:1727334209405676, mt_stat_.create_flush_dag_time:1727337670637045, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.678463] WDIAG [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:1250) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=0][errcode=-4184] sstable merge finish(ret=-4184, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752511928, storage_schema:{this:0x7f9268515330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:0, index_type:0, index_status:1, row_store_type:1, schema_version:1710828752511928, column_cnt:8, store_column_cnt:8, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:3, rowkey_array:[{column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}, {column_idx:20, meta_type:{type:"VARCHAR", collation:"utf8mb4_general_ci", coercibility:"INVALID"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:121}, table_count:1, [{i:0, table_key:{tablet_id:{id:121}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}, merge_scn:{val:1726241548806527000, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b8626bc0, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C779-0-0, dag_ret:0, dag_status:2, start_time:1727337670674240, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}, scanned_row_cnt_arr:0x7f9268515480, output_block_cnt_arr:0x7f9268515488, concurrent_cnt:1, estimate_row_cnt:1, estimate_occupy_size:0, latest_update_ts:1727337670674329, estimated_finish_time:1727337687675403}, compaction_filter:NULL, time_guard:total=2.02ms, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, task={this:0x7f92665d6250, type:16, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C779-0-0, dag_ret:0, dag_status:2, start_time:1727337670674240, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:10.678507] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670678305, end_time=1727337670678507, runtime=202, *this={this:0x7f9264c80030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77D-0-0, dag_ret:0, dag_status:2, start_time:1727337670675318, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.678513] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670676278, end_time=1727337670678513, runtime=2235, *this={this:0x7f92665d6250, type:16, status:2, dag:{ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C779-0-0, dag_ret:0, dag_status:2, start_time:1727337670674240, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}}) [2024-09-26 16:01:10.678520] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=3201, dag_cnt=2, dag_cnts_[dag.get_type()]=2, &dag=0x7f9264c46020, dag={ObIDag:{this:0x7f9264c46020, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C77D-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670675318, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:-8097406237500329300}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101015}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726211867106157000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.678530] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2855][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C77D-0-0] [lt=8] succeed to del sys task(removed_task={start_time:1727337670675313, task_id:YB42C0A8002D-0006230056D8C77D-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101015;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.678539] INFO [STORAGE.TRANS] get_schema_info (ob_memtable.cpp:2393) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=3] column cnt or schema version is updated by memtable(this={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337670637185, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}, max_column_cnt_on_memtable=0, max_schema_version_on_memtable=0) [2024-09-26 16:01:10.678563] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=4322, dag_cnt=1, dag_cnts_[dag.get_type()]=1, &dag=0x7f9264c44b60, dag={ObIDag:{this:0x7f9264c44b60, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C779-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670674240, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:8944614441376604790}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:121}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726178585269925002, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241548806527000, v:0}}}}) [2024-09-26 16:01:10.678558] INFO [STORAGE.COMPACTION] get_storage_schema_to_merge (ob_tablet_merge_ctx.cpp:1110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=17] get storage schema to merge(param={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, schema_ctx={schema_version:1710828752999056, storage_schema:{this:0x7f9262c6f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, use_schema_on_tablet=true, max_column_cnt_in_memtable=6, max_schema_version_in_memtable=1710828752999056, max_column_cnt_on_recorder=0) [2024-09-26 16:01:10.678571] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2850][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C779-0-0] [lt=6] succeed to del sys task(removed_task={start_time:1727337670674237, task_id:YB42C0A8002D-0006230056D8C779-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=121;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.678580] INFO [STORAGE] inner_loop_find_level_ (ob_query_engine.cpp:568) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=11] finish find split level(ret=-4018, ret="OB_ENTRY_NOT_EXIST", top_level=4, btree_node_count=0, total_rows=0) [2024-09-26 16:01:10.678600] INFO [STORAGE] init (ob_partition_parallel_merge_ctx.cpp:110) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=3] Succ to init parallel merge ctx(enable_parallel_minor_merge=true, tablet_size=134217728, merge_ctx.param_={merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}) [2024-09-26 16:01:10.678607] INFO [STORAGE] init_as_index (ob_macro_block.cpp:234) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=4] success to init data desc as index(ret=0, merge_schema={this:0x7f9262c6f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}, this={ls_id:{id:1}, tablet_id:{id:101067}, micro_block_size:16384, micro_block_size_limit:2095976, row_column_count:9, rowkey_column_count:8, schema_rowkey_col_cnt:6, full_stored_col_cnt:8, row_store_type:0, encoder_opt:{enable_bit_packing:false, store_sorted_var_len_numbers_dict:false, enable_raw:false, enable_dict:false, enable_int_diff:false, enable_str_diff:false, enable_hex_pack:false, enable_rle:false, enable_const:false}, compressor_type:1, schema_version:1710828752999056, merge_info:null, merge_type:3, snapshot_version:1726241132764173999, end_scn:{val:1726241132764173999, v:0}, need_prebuild_bloomfilter:false, bloomfilter_rowkey_prefix:0, encrypt_id:0, master_key_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", major_working_cluster_version:0, sstable_index_builder:null, is_ddl:false, col_desc_array:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=25 {type:"VARCHAR", collation:"binary", coercibility:"INVALID"} order=-1], default_col_checksum_array_valid:false, col_default_checksum_array:[0, 0, 0, 0, 0, 0, 0, 0]}) [2024-09-26 16:01:10.678639] INFO [STORAGE.COMPACTION] build_merge_ctx (ob_tablet_merge_task.cpp:1072) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=26] succeed to build merge ctx(tablet_id={id:101067}, ctx={param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, create_snapshot_version:0, is_full_merge:false, merge_level:0, progressive_merge_num:0, parallel_merge_ctx:{parallel_type:3, range_array:[{start_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MIN_OBJ,]store_rowkey:MIN}, end_key:{datum_cnt:1, group_idx:0, hash:0, [idx=0:MAX_OBJ,]store_rowkey:MAX}, group_idx:0, border_flag:{flag:0}}], concurrent_cnt:1, is_inited:true}, schema_ctx:{schema_version:1710828752999056, storage_schema:{this:0x7f9262c6f330, storage_schema_version:2, version:0, is_use_bloomfilter:0, column_info_simplified:1, compat_mode:0, table_type:5, index_type:1, index_status:1, row_store_type:1, schema_version:1710828752999056, column_cnt:6, store_column_cnt:6, tablet_size:134217728, pctfree:10, block_size:16384, progressive_merge_round:1, master_key_id:18446744073709551615, compressor_type:1, encryption:"", encrypt_key:"", rowkey_cnt:6, rowkey_array:[{column_idx:21, meta_type:{type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:16, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:17, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:18, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:19, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}, {column_idx:20, meta_type:{type:"BIGINT", collation:"binary", coercibility:"NUMERIC"}, order:0}], column_array:[]}}, tables_handle count:1, progressive_merge_round:0, progressive_merge_step:0, tables_handle:{tablet_id:{id:101067}, table_count:1, [{i:0, table_key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref:3}]}, schedule_major:false, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}, merge_scn:{val:1726241290462324999, v:0}, read_base_version:0, ls_handle:{ls_map_:0x7f92b7724040, ls_:0x7f92a6c04150, mod_:1}, tablet_handle:{obj:0x7f92b86a2d40, obj_pool:0x7f92cfbeb670, allocator:null, wash_priority:0, allow_copy_and_assign:true}, merge_progress:{is_inited:false, merge_dag:NULL, scanned_row_cnt_arr:null, output_block_cnt_arr:null, concurrent_cnt:0, estimate_row_cnt:0, estimate_occupy_size:0, latest_update_ts:1727337670677876, estimated_finish_time:0}, compaction_filter:NULL, time_guard:total=729us, rebuild_seq:0, data_version:0, need_parallel_minor_merge:true}, skip_rest_operation=false) [2024-09-26 16:01:10.678672] INFO [STORAGE.COMPACTION] prepare_merge_progress (ob_tablet_merge_ctx.cpp:1147) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=19] succeed to init merge progress(ret=0, merge_progress_={is_inited:true, merge_dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C785-0-0, dag_ret:0, dag_status:2, start_time:1727337670677846, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}, scanned_row_cnt_arr:0x7f9262c6f4a0, output_block_cnt_arr:0x7f9262c6f4a8, concurrent_cnt:1, estimate_row_cnt:227, estimate_occupy_size:0, latest_update_ts:1727337670677876, estimated_finish_time:1727337687678671}) [2024-09-26 16:01:10.678679] INFO [STORAGE.COMPACTION] process (ob_tablet_merge_task.cpp:991) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=6] succeed to init merge ctx(task={this:0x7f9264c48d60, type:15, status:2, dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C785-0-0, dag_ret:0, dag_status:2, start_time:1727337670677846, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.678684] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2860][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=4] task finish process(ret=0, start_time=1727337670677873, end_time=1727337670678684, runtime=811, *this={this:0x7f9264c48d60, type:15, status:2, dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C785-0-0, dag_ret:0, dag_status:2, start_time:1727337670677846, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.678693] INFO [COMMON] schedule_one (ob_dag_scheduler.cpp:3045) [2891][T1_DagScheduler][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=1] schedule one task(task=0x7f926652a030, priority="PRIO_COMPACTION_HIGH", group id=0, total_running_task_cnt=1, running_task_cnts_[priority]=1, low_limits_[priority]=6, up_limits_[priority]=6, task->get_dag()->get_dag_net()=NULL) [2024-09-26 16:01:10.678734] INFO [STORAGE.TRANS] init (ob_memtable_iterator.cpp:812) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=3] multi version scan iterator init succ(param.table_id_=101067, range=0x7f92a9f52200, read_info_={is_inited:true, compat_version:1, is_oracle_mode:false, schema_column_count:6, schema_rowkey_cnt:6, rowkey_cnt:8, cols_index:{rowkey_mode:true, for_memtable:false, schema_rowkey_cnt:6, column_cnt:8}, cols_desc:{ObIArray:[column_id=21 {type:"TIMESTAMP", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=16 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=17 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=18 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=19 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=20 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=7 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0, column_id=8 {type:"BIGINT", collation:"binary", coercibility:"NUMERIC"} order=0], data:0x7f92664ee060, count:8, allocator:0x7f92ba7fc7d8, capacity:8, init_cnt:8}, datum_utils:{is_oracle_mode:false, rowkey_cnt:8, col_cnt:8, is_inited:true, is_oracle_mode:false}}, row_={row_flag:{flag:"NOT_EXIST", flag_type:0}, trans_id:{txid:0}, scan_index:0, mvcc_row_flag:{first:0, uncommitted:0, shadow:0, compact:0, ghost:0, last:0, reserved:0, flag:0}, snapshot_version:0, fast_filter_skipped:false, have_uncommited_row:false, group_idx:0, count:8, datum_buffer:{capacity:32, datums:0x7f92664ee3e0, local_datums:0x7f92664ee3e0}, datums:[col_id=0:NOP,col_id=1:NOP,col_id=2:NOP,col_id=3:NOP,col_id=4:NOP,col_id=5:NOP,col_id=6:NOP,col_id=7:NOP,]}) [2024-09-26 16:01:10.678930] ERROR issue_dba_error (ob_log.cpp:1875) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4184, file="ob_block_manager.cpp", line_no=302, info="Failed to alloc block from io device") [2024-09-26 16:01:10.678932] EDIAG [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:302) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=2][errcode=-4184] Failed to alloc block from io device(ret=-4184) BACKTRACE:0x121f1f9c 0x53d75d5 0x538fd5d 0x538f79f 0x538f6dc 0x538f50d 0xf44a67c 0xf449fec 0xf4f9b33 0xf4f1832 0xf4f7ae7 0xf4ffce3 0xf480613 0xf202cf5 0xf211025 0xf213463 0xf260914 0xfd75eea 0x53af0a1 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83 [2024-09-26 16:01:10.678940] INFO [STORAGE.COMPACTION] reset (ob_partition_rows_merger.cpp:912) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=0] partition merge iter row count(i=0, row_count=227, ghost_row_count=0, pkey={tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, table={ObITable:{this:0x7f92a3bcbd80, key:{tablet_id:{id:101067}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}, ref_cnt:3, upper_trans_version:9223372036854775807, timestamp:1727334208147168}, this:0x7f92a3bcbd80, timestamp:1727334208147168, state:0, freeze_clock:1, max_schema_version:0, max_data_schema_version:1710828752999056, max_column_cnt:6, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:49, clock:603979776}, host:0x7f92ba380030, arena_handle:{allocated:0}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:false, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726241290462324999, v:0}, rec_scn:{val:1726236179227314000, v:0}, snapshot_version:{val:1726241132764173999, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334209405025, mt_stat_.ready_for_flush_time:1727334209405811, mt_stat_.create_flush_dag_time:1727337670637185, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}}) [2024-09-26 16:01:10.678957] INFO [COMMON] do_work (ob_dag_scheduler.cpp:248) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=0] task finish process(ret=-4184, start_time=1727337670678706, end_time=1727337670678957, runtime=251, *this={this:0x7f926652a030, type:1, status:2, dag:{ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C785-0-0, dag_ret:0, dag_status:2, start_time:1727337670677846, running_task_cnt:1, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}}) [2024-09-26 16:01:10.678966] INFO [COMMON] finish_dag_ (ob_dag_scheduler.cpp:2611) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=0] dag finished(dag_ret=-4184, runtime=1119, dag_cnt=0, dag_cnts_[dag.get_type()]=0, &dag=0x7f9264c489a0, dag={ObIDag:{this:0x7f9264c489a0, type:0, name:"MINI_MERGE", id:YB42C0A8002D-0006230056D8C785-0-0, dag_ret:-4184, dag_status:5, start_time:1727337670677846, running_task_cnt:0, indegree:0, consumer_group_id:0, hash:6699723950958743903}, param:{merge_type:"MINI_MERGE", merge_version:0, ls_id:{id:1}, tablet_id:{id:101067}, report_:null, for_diagnose:false, is_tenant_major_merge:false, need_swap_tablet_flag:false}, compat_mode:0, ctx:{sstable_version_range:{multi_version_start:1726149987022722000, base_version:0, snapshot_version:1726241132764173999}, scn_range:{start_scn:{val:1726211867305630000, v:0}, end_scn:{val:1726241290462324999, v:0}}}}) [2024-09-26 16:01:10.678972] INFO [SERVER] del_task (ob_sys_task_stat.cpp:172) [2878][T1_MINI_MERGE][T1][YB42C0A8002D-0006230056D8C785-0-0] [lt=4] succeed to del sys task(removed_task={start_time:1727337670677842, task_id:YB42C0A8002D-0006230056D8C785-0-0, task_type:3, svr_ip:"192.168.0.45:2882", tenant_id:1, is_cancel:false, comment:"info="DAG_MINI_MERGE";ls_id=1;tablet_id=101067;compaction_scn=0;extra_info="merge_type="MINI_MERGE"";"}) [2024-09-26 16:01:10.691245] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:10.691254] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.691723] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337670363680]}) [2024-09-26 16:01:10.721620] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:10.721627] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.736543] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:10.736565] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337670736540) [2024-09-26 16:01:10.736571] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337670636407, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:10.736588] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223399973012, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.736594] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:10.736602] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:10.736608] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:10.736611] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:10.736618] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.736620] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.736622] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337670736575) [2024-09-26 16:01:10.751722] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:10.751729] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.782178] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:10.782185] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.792170] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337670363680]}) [2024-09-26 16:01:10.805706] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:10.805810] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=53, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:10.806424] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1069538, clean_num=31457) [2024-09-26 16:01:10.810464] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:01:10.812439] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:10.812448] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.821220] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:10.821224] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.821251] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.821257] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.821262] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.821270] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.821274] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.821288] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.821304] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.822205] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.822266] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=8] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.822274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:10.823163] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=10][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337670363680]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:01:10.823173] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=9][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:01:10.837833] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:10.837855] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337670837829) [2024-09-26 16:01:10.837861] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337670736575, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:10.837866] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:10.837875] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:10.837878] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:10.838208] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578B-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:10.838268] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578B-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:10.838306] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337670838305, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=433, wlock_time=11, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:10.838315] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:10.838333] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223400074305, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.838338] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:10.838344] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:10.838350] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:10.838352] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:10.838366] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.838369] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.838370] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337670838320) [2024-09-26 16:01:10.842727] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:01:10.842735] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.864940] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:10.864948] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337670864938], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:10.864954] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337670864938], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:10.873419] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:10.873438] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.879852] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337671811144, now=[mts=1727337669881374]) [2024-09-26 16:01:10.879860] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337671811144, now=[mts=1727337669881374], now0=[mts=1727337669881374], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:10.879875] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=14][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a312eea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500885, assoc_session_id:3221500885, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337669879612, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:10.879892] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22f01f8, id:3221500885, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312eea0}) [2024-09-26 16:01:10.879900] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500885, *tx_desc={this:0x7f92a312eea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500885, assoc_session_id:3221500885, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337669879612, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:10.879908] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500885, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a312eea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500885, assoc_session_id:3221500885, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337669879612, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337669881350, use_das=false, nested_level=0, session={this:0x7f92a22f01f8, id:3221500885, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312eea0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:10.879925] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:10.879930] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:10.879933] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:10.879936] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:10.879939] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:10.879943] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:10.879947] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=1001197, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337671881144) [2024-09-26 16:01:10.879951] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:10.879959] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:10.879920] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:68, tid:2932}]) [2024-09-26 16:01:10.879961] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:10.879962] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:10.879968] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:10.880013] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:01:10.880017] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=4][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:10.892513] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337670864938]}) [2024-09-26 16:01:10.892508] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=4] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2024-09-26 16:01:10.892528] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:99) [2937][T1_TabletShellT][T1][Y0-0000000000000000-0-0] [lt=16] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=694) [2024-09-26 16:01:10.903556] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:10.903566] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.933603] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21] success to notify_need_writing_throttling True [2024-09-26 16:01:10.933614] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.937909] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:10.937926] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337670937906) [2024-09-26 16:01:10.937932] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337670837864, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:10.937953] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223400173794, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:10.937959] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:10.937967] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:10.937972] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:10.937975] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:10.937980] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.937983] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:10.937985] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337670937936) [2024-09-26 16:01:10.963722] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=27] success to notify_need_writing_throttling True [2024-09-26 16:01:10.963734] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.964189] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=9] PNIO [ratelimit] time: 1727337670964189, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007564, add_bytes: 0 [2024-09-26 16:01:10.992601] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337670864938]}) [2024-09-26 16:01:10.993798] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20] success to notify_need_writing_throttling True [2024-09-26 16:01:10.993806] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:10.995190] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:10.995628] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:203) [2925][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=12] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f92b77d02b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2024-09-26 16:01:11.006641] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:11.006762] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=51, refresh_score_time=15, wash_time=1) [2024-09-26 16:01:11.007391] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1100995, clean_num=31457) [2024-09-26 16:01:11.010577] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:01:11.021580] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185946-0-0] [lt=43][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:11.021589] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185946-0-0] [lt=8][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:11.021598] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185946-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:11.021603] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185946-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:11.023994] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:11.024000] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.027313] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:11.027326] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-09-26 16:01:11.027332] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=6][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:11.027337] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-09-26 16:01:11.027342] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:11.027345] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:11.027350] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-09-26 16:01:11.027354] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:11.027358] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:11.027361] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:11.027363] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=2][errcode=-5019] resolve table failed(ret=-5019) [2024-09-26 16:01:11.027366] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:2822) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=2][errcode=-5019] resolve joined table item failed(ret=-5019) [2024-09-26 16:01:11.027368] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2676) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=2][errcode=-5019] resolve joined table failed(ret=-5019) [2024-09-26 16:01:11.027371] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:11.027377] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:11.027381] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:11.027384] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:11.027392] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:11.027395] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:11.027400] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:11.027403] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:11.027407] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2024-09-26 16:01:11.027411] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:11.027414] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:11.027466] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=50][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:11.027491] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=22][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:11.027495] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:11.027498] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:11.027509] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=4][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:11.027513] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:11.027517] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [3024][BlackListServic][T0][YB42C0A8002D-0006230056284FA9-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2024-09-26 16:01:11.027522] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:01:11.027525] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:11.027529] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:11.027532] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337671027138, sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:01:11.027537] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:111) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:11.027541] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:222) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select /*+query_timeout(2000000)*/ a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2024-09-26 16:01:11.027573] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=4] ls blacklist refresh finish(cost_time=706) [2024-09-26 16:01:11.038009] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:11.038020] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337671038005) [2024-09-26 16:01:11.038027] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337670937935, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:11.038047] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223400273311, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:11.038054] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:11.038063] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:11.038069] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:11.038072] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:11.038079] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.038082] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.038085] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337671038031) [2024-09-26 16:01:11.053593] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500890, support_ssl=false) [2024-09-26 16:01:11.053625] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221500890, sess.client_addr_="127.0.0.1:36402") [2024-09-26 16:01:11.053631] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=4] accept one succ(*s={this:0x7f929e804030, session_id:3221500890, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:11.053677] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:11.053686] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2689][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=4] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:11.053723] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA9-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:11.053904] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA9-0-0] [lt=3][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:11.053911] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA9-0-0] [lt=6][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"QoYPe+F]"0iCRP|?d[yp"}, ret=-4043) [2024-09-26 16:01:11.053919] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA9-0-0] [lt=6][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:11.053922] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA9-0-0] [lt=2][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:11.053930] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA9-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:11.053965] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA9-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500890, proxy_sessid:0}) [2024-09-26 16:01:11.053970] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA9-0-0] [lt=5] mark session id unused(sessid=3221500890) [2024-09-26 16:01:11.053974] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA9-0-0] [lt=2][errcode=0] server close connection(sessid=3221500890, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:11.053977] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA9-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500890, proxy_sessid=0) [2024-09-26 16:01:11.053979] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA9-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:11.053983] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FA9-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500890, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:11.054016] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f929e804030, session_id:3221500890, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337671053073, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:11.054024] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221500890, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:11.054027] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=2] can close safely, do destroy(*s={this:0x7f929e804030, session_id:3221500890, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337671053073, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:11.054031] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2689][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500890, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:11.054162] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:11.054167] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.073370] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:11.073382] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:01:11.073387] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:01:11.073390] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:01:11.073393] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:11.073395] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:11.073399] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:01:11.073426] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:11.073430] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:11.073432] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:11.073434] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:11.073439] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:11.073442] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:11.073444] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:11.073450] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:11.073453] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:11.073456] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:11.073458] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:11.073479] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=21][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:01:11.073482] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:11.073486] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:11.073491] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:11.073497] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:11.073499] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:11.073501] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:11.073507] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:11.073510] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A0-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:11.073514] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A0-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:01:11.073517] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:11.073519] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:11.073521] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:11.073523] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337671073275, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:11.073527] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A0-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:11.073530] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:01:11.073565] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A0-0-0] [lt=2][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:01:11.073568] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A0-0-0] [lt=3][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:01:11.078115] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=8] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:11.078475] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=4] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:11.078485] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:11.078489] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:11.078492] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:11.078495] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:11.078497] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:11.078539] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=40][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:11.078541] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:11.078543] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:11.078545] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:11.078546] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:11.078548] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:11.078571] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=23][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:11.078574] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:11.078579] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:11.078581] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:11.078583] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:11.078586] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:11.078588] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:01:11.078590] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:11.078595] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=4][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:11.078600] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:11.078604] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:11.078607] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:11.078608] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:11.078613] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:11.078617] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E7E-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:11.078620] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2579][RSAsyncTask0][T0][YB42C0A8002D-0006230057584E7E-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:01:11.078623] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:11.078625] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:11.078627] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:11.078630] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337671078419, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:11.078633] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:11.078684] WDIAG [SHARE] run2 (ob_async_task_queue.cpp:150) [2579][RSAsyncTask0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] task process failed, start retry(max retry time=0, retry interval=1000000, ret=-5019) [2024-09-26 16:01:11.084262] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.084270] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.092686] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337670864938]}) [2024-09-26 16:01:11.092697] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=7] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:11.114357] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:11.114368] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.138252] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:11.138266] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:11.138274] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337671138248) [2024-09-26 16:01:11.138280] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337671038031, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:11.138298] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223400374375, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:11.138304] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:11.138313] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:11.138318] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:11.138321] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:11.138327] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.138329] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.138332] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337671138282) [2024-09-26 16:01:11.144423] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:01:11.144439] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.148579] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:221) [2596][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=10] [ASYNC TASK QUEUE](queue_size=0) [2024-09-26 16:01:11.174514] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:11.174525] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.192946] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337670864938]}) [2024-09-26 16:01:11.193296] INFO [COMMON] print_io_status (ob_io_manager.cpp:1384) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1] [IO STATUS SYS](tenant_id=1, io_status=sys_group_name: SHARED_BLOCK_ReaderWriter_IO, mode: read, size: 4096.00, iops: 52.89, rt: 2592.09) [2024-09-26 16:01:11.193308] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CONFIG](tenant_id=1, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:11.193335] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 13298.00) [2024-09-26 16:01:11.193338] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:11.193345] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=5] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:11.193349] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=3] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:11.204621] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:01:11.204629] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.207662] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:11.207776] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=51, refresh_score_time=16, wash_time=2) [2024-09-26 16:01:11.208393] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1132452, clean_num=31457) [2024-09-26 16:01:11.210962] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=975136, replace_num=15728) [2024-09-26 16:01:11.210969] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:01:11.214138] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:01:11.214152] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=9] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:11.214158] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:01:11.234668] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:11.234676] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.238331] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:11.238342] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337671238328) [2024-09-26 16:01:11.238347] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337671138282, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:11.238366] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223400474058, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:11.238372] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:11.238381] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:11.238386] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:11.238389] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:11.238396] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.238398] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.238400] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337671238350) [2024-09-26 16:01:11.264742] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:11.264750] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.264760] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:11.266886] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:11.266897] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:11.266901] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:11.266906] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:11.266909] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:11.266911] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:11.266915] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:11.266917] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:11.266919] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:11.266921] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:11.266923] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:11.266926] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:11.266928] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:11.266930] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:11.266937] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:11.266940] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:11.266943] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:11.266945] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:11.266947] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:01:11.266950] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=3][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:11.266952] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:11.266958] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:11.266964] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:11.266984] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:11.266986] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:11.266992] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:11.266996] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:11.266998] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:01:11.267001] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:11.267003] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:11.267005] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:11.267007] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337671266813, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:11.267011] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:11.267014] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:11.267044] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:11.267047] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:01:11.267049] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:11.267051] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:11.267054] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=1][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:11.267057] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=2][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:11.267058] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858AF-0-0] [lt=1][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:11.278615] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=12][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337670864938]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:01:11.278626] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=10][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:01:11.292989] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337670864938]}) [2024-09-26 16:01:11.293004] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8DA0E-0-0] [lt=9][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337671292983}) [2024-09-26 16:01:11.294861] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:01:11.294868] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.301751] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=8][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337670864938]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:01:11.301759] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=8][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:01:11.324961] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:11.324968] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.329237] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=10][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337672258346, now=[mts=1727337670328649]) [2024-09-26 16:01:11.329244] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337672258346, now=[mts=1727337670328649], now0=[mts=1727337670328649], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:11.329250] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92690188c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337670328025, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:11.329267] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92690188c0}) [2024-09-26 16:01:11.329275] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92690188c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337670328025, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:11.329282] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=6] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92690188c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337670328025, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337670328620, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92690188c0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:11.329299] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:11.329304] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:11.329307] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:11.329310] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:11.329312] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:11.329316] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:11.329321] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=999025, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337672328346) [2024-09-26 16:01:11.329324] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:11.329344] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:11.329346] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:11.329348] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:11.329354] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:11.329390] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=58) [2024-09-26 16:01:11.329393] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:11.338431] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:11.338439] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337671338428) [2024-09-26 16:01:11.338443] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337671238350, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:11.338448] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:11.338456] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:11.338459] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:11.338792] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:11.338808] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=15][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:11.338816] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=8][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:11.338820] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=4][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:11.338824] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:11.338826] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:11.338829] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:11.338832] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:11.338834] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:11.338836] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:11.338838] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:11.338841] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:11.338844] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:11.338846] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:11.338853] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:11.338856] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:11.338859] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:11.338864] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:11.338866] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:11.338869] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:11.338871] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:11.338876] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:11.338881] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:11.338884] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:11.338885] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:11.338890] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:11.338893] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:11.338896] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578C-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:11.338912] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:11.338916] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:11.338919] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:11.338922] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8be0, start=1727337671338722, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:11.338925] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:11.338930] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:11.338957] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:11.338960] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337671338959, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=506, wlock_time=8, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:11.338966] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:11.338972] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:11.338989] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223400574996, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:11.338994] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:11.339000] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:11.339005] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:11.339008] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:11.339013] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.339016] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.339018] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337671338975) [2024-09-26 16:01:11.354995] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.355001] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.365388] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.365398] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337671365385], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:11.365404] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337671365385], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:11.365408] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=6945, get_gts_with_stc_cnt=12094, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2024-09-26 16:01:11.365415] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337671365385]}) [2024-09-26 16:01:11.365420] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:11.365423] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:11.385302] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.385309] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.393153] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337671365385]}) [2024-09-26 16:01:11.393889] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=9] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:11.408708] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:11.408817] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Wash time detail, (compute_wash_size_time=59, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:11.409430] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1163909, clean_num=31457) [2024-09-26 16:01:11.411035] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:01:11.415352] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.415358] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.435756] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337700364812, now=[mts=1727337670434843]) [2024-09-26 16:01:11.435765] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337700364812, now=[mts=1727337670434843], now0=[mts=1727337670434843], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:11.435772] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926902cea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337670434463, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:11.435790] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f9269128038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926902cea0}) [2024-09-26 16:01:11.435799] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=9][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926902cea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337670434463, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:11.435806] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c56d2378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926902cea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337670434463, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337670434817, use_das=false, nested_level=0, session={this:0x7f9269128038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926902cea0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:11.435823] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:11.435828] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:11.435831] WDIAG [SQL] open (ob_result_set.cpp:157) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:11.435834] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:11.435837] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:01:11.435841] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:11.435845] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3] already timeout, do not need sleep(sleep_us=0, remain_us=28998967, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337700434812) [2024-09-26 16:01:11.435849] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:11.435856] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:11.435858] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:11.435860] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:11.435865] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:11.435904] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=74) [2024-09-26 16:01:11.435907] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:11.435950] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] ====== tenant freeze timer task ====== [2024-09-26 16:01:11.435957] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:01:11.435988] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337670636259, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337670636210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337670636278, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:01:11.436010] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:01:11.436047] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E8-0-0] [lt=4] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:01:11.436058] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E8-0-0] [lt=8] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:01:11.436064] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E8-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:01:11.436067] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E8-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:01:11.436071] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E8-0-0] [lt=3][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:01:11.436074] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E8-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:01:11.436076] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E8-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:01:11.436087] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E8-0-0] [lt=11] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:01:11.436090] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E8-0-0] [lt=3][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:11.436092] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E8-0-0] [lt=2] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:11.436095] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E8-0-0] [lt=1][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:11.436134] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851E8-0-0] [lt=5] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:01:11.438462] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:11.438477] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337671438459) [2024-09-26 16:01:11.438482] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337671338445, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:11.438498] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223400673957, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:11.438503] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:11.438510] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:11.438514] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:11.438517] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:11.438523] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.438525] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.438527] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337671438484) [2024-09-26 16:01:11.445412] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:11.445418] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.467602] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337671365385]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:01:11.467613] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=10][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:01:11.469968] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.469975] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.469979] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.469969] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=7] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.469984] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.469983] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.469986] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.469988] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.469988] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.470493] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.470509] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=16][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.470512] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.470515] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.470517] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.470513] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D4635-0-0] [lt=8][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:11.470519] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.470529] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.470531] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.470534] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.471111] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.471112] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.471114] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.471115] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.471116] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.471117] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.471120] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.471122] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.471126] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.471645] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.471650] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.471652] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.471656] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.471658] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.471659] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.471647] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.471665] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=17][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.471669] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.472442] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.472448] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.472450] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.472453] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.472466] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.472468] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.472477] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.472480] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.472482] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.473034] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.473037] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.473038] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.473039] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:11.473042] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.473634] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.473635] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.473639] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.474145] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.474235] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.474237] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.474832] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.474839] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.474834] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.475391] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.475398] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.475460] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.475510] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.475514] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=3][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.475975] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.475978] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.475982] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.476526] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.476562] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.476567] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.477121] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.477249] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.477255] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.477650] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.477879] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.477884] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.478511] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.478523] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.478527] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.479092] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.479093] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.479097] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.479659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.479659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.479663] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.480235] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.480235] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.480240] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.480784] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.480787] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.480791] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.481368] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.481383] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.481415] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.481961] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.481969] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=13][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.482020] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.482506] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.482512] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.482649] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.483151] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.483157] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.483292] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.483674] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.483680] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.483956] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.484256] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.484262] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.484478] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.484857] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.484863] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.485192] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.485539] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.485545] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.485773] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.486114] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.486120] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.486338] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.486668] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.486672] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.486908] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.487280] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.487287] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.487500] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.487853] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.487859] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.488117] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.488410] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.488414] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.488696] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.488977] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.488984] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.489230] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.489553] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:11.493169] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337671365385]}) [2024-09-26 16:01:11.506906] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.506914] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.517092] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1727337671517091, bytes: 14746687, bw: 0.004029 MB/s, add_ts: 1007645, add_bytes: 4257 [2024-09-26 16:01:11.527293] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185947-0-0] [lt=34][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:11.527313] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185947-0-0] [lt=18][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:11.527324] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185947-0-0] [lt=10][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:11.527330] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185947-0-0] [lt=6][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:11.536979] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.536986] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.538786] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:11.538797] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337671538782) [2024-09-26 16:01:11.538802] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337671438484, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:11.538822] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223400775139, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:11.538829] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:11.538837] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:11.538842] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:11.538845] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:11.538852] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.538854] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.538856] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337671538806) [2024-09-26 16:01:11.551634] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6717, mem_used=12478464) [2024-09-26 16:01:11.567099] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.567106] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.568511] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:01:11.593229] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337671365385]}) [2024-09-26 16:01:11.597271] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:11.597277] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.602198] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.602220] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.602228] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=1] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.602260] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.609571] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:11.609675] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=53, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:11.610288] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1195366, clean_num=31457) [2024-09-26 16:01:11.611087] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:01:11.627310] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.627317] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.632260] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:11.638889] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:11.638900] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337671638886) [2024-09-26 16:01:11.638923] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337671538806, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:11.638942] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223400874794, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:11.638948] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:11.638973] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:11.638978] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:11.638981] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:11.638988] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.638990] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.638992] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337671638927) [2024-09-26 16:01:11.657753] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.657762] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.667707] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=5] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:11.680781] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4184, dropped:500, tid:2860}]) [2024-09-26 16:01:11.687665] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=16] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:01:11.687852] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.687859] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.693400] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337671365385]}) [2024-09-26 16:01:11.717982] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.717991] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.739050] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:11.739064] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337671739046) [2024-09-26 16:01:11.739070] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337671638926, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:11.739089] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223400975133, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:11.739095] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:11.739103] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:11.739108] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:11.739111] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:11.739117] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.739119] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.739121] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337671739073) [2024-09-26 16:01:11.748114] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:11.748124] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.778249] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:11.778260] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.793452] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337671365385]}) [2024-09-26 16:01:11.808324] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:11.808337] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.810485] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:11.810606] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=74, refresh_score_time=17, wash_time=1) [2024-09-26 16:01:11.811167] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:01:11.811227] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1226823, clean_num=31457) [2024-09-26 16:01:11.820601] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.821108] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=13] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.821172] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.821497] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.821505] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.821512] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.821517] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=12] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.821972] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:11.821988] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.821994] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.822191] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.822203] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=6] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:11.830275] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [2915][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=8] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2024-09-26 16:01:11.838445] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:01:11.838454] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.839092] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:11.839105] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337671839089) [2024-09-26 16:01:11.839111] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337671739073, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:11.839117] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:11.839128] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:11.839131] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:11.839484] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578D-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:11.839496] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578D-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:11.839501] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578D-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:11.839504] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578D-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:11.839508] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578D-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:11.839510] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578D-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:11.839513] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578D-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:11.839516] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578D-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:11.839539] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578D-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:11.839575] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337671839574, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=451, wlock_time=9, check_leader_time=1, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:11.839587] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:11.839606] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] get wrs ts(ls_id={id:1}, delta=223401075031, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:11.839633] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:11.839640] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:11.839645] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:11.839647] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:11.839654] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.839656] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.839657] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337671839592) [2024-09-26 16:01:11.865816] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:11.865833] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337671865813], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:11.865842] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337671865813], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:11.868538] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:11.868547] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.881024] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=8][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337671811144, now=[mts=1727337670880012]) [2024-09-26 16:01:11.881038] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=13][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337671811144, now=[mts=1727337670880012], now0=[mts=1727337670880012], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:11.881045] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92a312eea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500885, assoc_session_id:3221500885, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337670879847, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:11.881063] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22f01f8, id:3221500885, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312eea0}) [2024-09-26 16:01:11.881071] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=3221500885, *tx_desc={this:0x7f92a312eea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500885, assoc_session_id:3221500885, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337670879847, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:11.881078] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=3221500885, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a312eea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500885, assoc_session_id:3221500885, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337670879847, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337669881350, use_das=false, nested_level=0, session={this:0x7f92a22f01f8, id:3221500885, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312eea0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:11.881096] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:11.881100] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:11.881103] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:11.881105] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:11.881108] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:11.881112] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:11.881116] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3] will sleep(sleep_us=28, remain_us=28, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4283, timeout_timestamp=1727337671881144) [2024-09-26 16:01:11.881254] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:11.881263] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=5][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:11.881265] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:11.881267] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:11.881272] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:11.881311] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1764) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4012] get gts timeout(ret=-4012, expire_ts=1727337671811144, current_time=1727337671880912) [2024-09-26 16:01:11.881314] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=3][errcode=-4012] acquire global snapshot fail(ret=-4012, gts_ahead=0, expire_ts=1727337671811144, now=[mts=1727337671881311], now0=[mts=1727337671881311], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:11.881318] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=4][errcode=-4012] acquire global snapshot fail(ret=-4012, tx={this:0x7f92a312eea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500885, assoc_session_id:3221500885, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337671880912, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:11.881326] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=8][errcode=-4012] fail to get snapshot(ret=-4012, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f92a22f01f8, id:3221500885, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312eea0}) [2024-09-26 16:01:11.881331] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=5][errcode=-4012] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4012, session_id=3221500885, *tx_desc={this:0x7f92a312eea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500885, assoc_session_id:3221500885, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337671880912, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:11.881336] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=5] start stmt(ret=-4012, auto_commit=true, session_id=3221500885, snapshot={this:0x7f92a96543f8, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92a312eea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:3221500885, assoc_session_id:3221500885, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337671880912, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337669881350, use_das=false, nested_level=0, session={this:0x7f92a22f01f8, id:3221500885, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:0x7f92a312eea0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:11.881347] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=11][errcode=-4012] fail to start stmt(ret=-4012, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:11.881349] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4012] fail start stmt(ret=-4012) [2024-09-26 16:01:11.881351] WDIAG [SQL] open (ob_result_set.cpp:157) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4012] execute plan failed(ret=-4012) [2024-09-26 16:01:11.881352] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4012] open result set failed(ret=-4012) [2024-09-26 16:01:11.881354] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:11.881356] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4012] execute failed(ret=-4012, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:11.881358] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:01:11.881363] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=4][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:01:11.881365] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4012] result set close failed(ret=-4012) [2024-09-26 16:01:11.881366] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2024-09-26 16:01:11.881368] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=1][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:01:11.881370] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000020) [2024-09-26 16:01:11.881373] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-000623005688508F-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:01:11.881376] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:11.881379] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:01:11.881381] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:01:11.881384] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] query failed(ret=-4012, conn=0x7f92cefa8340, start=1727337669881346, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:11.881387] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] read failed(ret=-4012) [2024-09-26 16:01:11.881390] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:11.881431] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:01:11.881435] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:01:11.881437] WDIAG [SHARE] get_snapshot_gc_scn (ob_global_stat_proxy.cpp:164) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:01:11.881439] WDIAG [STORAGE] get_global_info (ob_tenant_freeze_info_mgr.cpp:811) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to get global info(ret=-4012, tenant_id=1) [2024-09-26 16:01:11.881442] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:954) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] failed to get global info(ret=-4012) [2024-09-26 16:01:11.881444] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1008) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4012] fail to try update info(tmp_ret=-4012, tmp_ret="OB_TIMEOUT") [2024-09-26 16:01:11.881449] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1044) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2] success to update min reserved snapshot(reserved_snapshot=0, duration=1800, snapshot_gc_ts_=0) [2024-09-26 16:01:11.881453] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1071) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4] update reserved snapshot finished(cost_ts=5, reserved_snapshot=0) [2024-09-26 16:01:11.893617] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=41] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337671865813]}) [2024-09-26 16:01:11.898704] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:11.898714] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.929003] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:11.929017] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.939248] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:11.939272] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337671939244) [2024-09-26 16:01:11.939279] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337671839114, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:11.939299] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223401174780, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:11.939307] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:11.939317] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:11.939322] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:11.939325] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:11.939332] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.939335] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:11.939337] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337671939282) [2024-09-26 16:01:11.960380] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:11.960393] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.971849] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=8] PNIO [ratelimit] time: 1727337671971846, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007657, add_bytes: 0 [2024-09-26 16:01:11.990550] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=13] success to notify_need_writing_throttling True [2024-09-26 16:01:11.990583] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:11.994063] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337671865813]}) [2024-09-26 16:01:11.995824] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:104) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] tx gc loop thread is running(MTL_ID()=1) [2024-09-26 16:01:11.995835] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:111) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] try gc retain ctx [2024-09-26 16:01:11.996278] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:232) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=3] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1726163586617842000, v:0}, clog_base_lsn:{lsn:627966418944}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1727114270763524000, v:0}, all_id_meta:{id_meta:[{limited_id:1727114284339921000, latest_log_ts:{val:1727114267745234001, v:0}}, {limited_id:188000001, latest_log_ts:{val:1727111170115172004, v:0}}, {limited_id:880000001, latest_log_ts:{val:1727109559140843003, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:0, log_handler:{role:1, proposal_id:374, palf_env_:0x7f92b51e0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2024-09-26 16:01:11.996307] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:12.011315] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:01:12.011476] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:12.011648] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash time detail, (compute_wash_size_time=108, refresh_score_time=17, wash_time=2) [2024-09-26 16:01:12.012518] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1258280, clean_num=31457) [2024-09-26 16:01:12.020791] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=17] success to notify_need_writing_throttling True [2024-09-26 16:01:12.020801] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.022249] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185948-0-0] [lt=40][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:12.022260] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185948-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:12.022268] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185948-0-0] [lt=7][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:12.022273] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185948-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:12.039300] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:12.039329] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337672039296) [2024-09-26 16:01:12.039336] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337671939282, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:12.039380] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223401275138, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:12.039403] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:12.039411] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:12.039417] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:12.039423] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:12.039439] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.039442] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.039444] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337672039339) [2024-09-26 16:01:12.050876] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:12.050886] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.054489] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500894, support_ssl=false) [2024-09-26 16:01:12.054509] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221500894, sess.client_addr_="127.0.0.1:36404") [2024-09-26 16:01:12.054513] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f929e804d70, session_id:3221500894, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:143}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:12.054583] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:12.054596] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=5] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:12.054640] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA9-0-0] [lt=7] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:12.054841] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA9-0-0] [lt=15][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:12.054847] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA9-0-0] [lt=5][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"}gliIr#RV9W.}OO}I4e#"}, ret=-4043) [2024-09-26 16:01:12.054854] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA9-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:12.054858] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA9-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:12.054867] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA9-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:12.054903] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA9-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500894, proxy_sessid:0}) [2024-09-26 16:01:12.054908] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA9-0-0] [lt=4] mark session id unused(sessid=3221500894) [2024-09-26 16:01:12.054912] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA9-0-0] [lt=2][errcode=0] server close connection(sessid=3221500894, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:12.054915] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA9-0-0] [lt=2][errcode=-4018] get session fail(ret=-4018, sessid=3221500894, proxy_sessid=0) [2024-09-26 16:01:12.054917] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA9-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:12.054920] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FA9-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500894, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:12.055009] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4015] close sql sock by user req(*s={this:0x7f929e804d70, session_id:3221500894, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337672054407, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:12.055019] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] kill and revert session(conn.sessid_=3221500894, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:12.055022] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f929e804d70, session_id:3221500894, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:143}, err:5, last_decode_time:1727337672054407, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:12.055026] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500894, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:12.074051] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A1-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:12.074064] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A1-0-0] [lt=12][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:12.074095] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A1-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:12.078711] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=8] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:12.079018] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E80-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:12.079028] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E80-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:12.079047] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E80-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:12.081029] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:12.081037] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.081238] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=10][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:100, tid:3024}]) [2024-09-26 16:01:12.092807] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=7] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:12.094208] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337671865813]}) [2024-09-26 16:01:12.111146] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.111153] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.139378] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:12.139388] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:12.139395] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337672139376) [2024-09-26 16:01:12.139400] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337672039339, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:12.139418] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223401375026, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:12.139423] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:12.139431] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:12.139447] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:12.139450] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:12.139457] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.139460] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.139462] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337672139403) [2024-09-26 16:01:12.141268] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.141275] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.171314] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:12.171321] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.194671] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337671865813]}) [2024-09-26 16:01:12.194885] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 10649.00) [2024-09-26 16:01:12.194896] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:12.194908] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:12.194912] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:12.201403] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:12.201410] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.204730] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=4] ====== check clog disk timer task ====== [2024-09-26 16:01:12.204736] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=5] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:01:12.204744] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:01:12.211367] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:01:12.212700] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:12.212820] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash time detail, (compute_wash_size_time=65, refresh_score_time=18, wash_time=1) [2024-09-26 16:01:12.213484] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1289737, clean_num=31457) [2024-09-26 16:01:12.231516] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.231523] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.239472] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:12.239485] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337672239469) [2024-09-26 16:01:12.239491] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337672139403, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:12.239510] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223401475430, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:12.239522] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:12.239530] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:12.239536] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:12.239539] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:12.239545] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.239548] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.239549] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337672239494) [2024-09-26 16:01:12.261583] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:12.261591] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.267558] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:12.267568] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:12.267573] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:12.267576] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:12.267579] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:12.267581] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:12.267584] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:12.267586] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:12.267589] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:12.267591] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:12.267593] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:12.267595] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:12.267597] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:12.267599] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:12.267606] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:12.267608] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:12.267610] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:12.267613] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:12.267615] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2024-09-26 16:01:12.267617] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:12.267620] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=3][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:12.267625] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=3][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:12.267631] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:12.267633] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:12.267634] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:12.267639] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:12.267649] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:12.267653] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2024-09-26 16:01:12.267656] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:12.267658] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:12.267660] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:12.267663] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337672267487, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:12.267666] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:12.267669] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2024-09-26 16:01:12.267697] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:12.267699] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2024-09-26 16:01:12.267702] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:12.267704] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:12.267707] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:12.267709] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:194) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=1][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2024-09-26 16:01:12.267711] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B0-0-0] [lt=2][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:12.291689] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:12.291699] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.291709] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:12.295104] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337671865813]}) [2024-09-26 16:01:12.295123] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8DA1A-0-0] [lt=7][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337672295098}) [2024-09-26 16:01:12.321795] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:12.321805] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.329487] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337672258346, now=[mts=1727337671329389]) [2024-09-26 16:01:12.329500] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=12][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337672258346, now=[mts=1727337671329389], now0=[mts=1727337671329389], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:12.329507] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f92690188c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337671328831, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:12.329524] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92690188c0}) [2024-09-26 16:01:12.329533] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f92690188c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337671328831, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:12.329540] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c66d2998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f92690188c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337671328831, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337670328620, use_das=false, nested_level=0, session={this:0x7f928657a038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f92690188c0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:12.329558] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:12.329563] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:12.329566] WDIAG [SQL] open (ob_result_set.cpp:157) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:12.329569] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:12.329572] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:12.329576] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:12.329580] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:01:12.329587] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:12.329589] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:12.329591] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:01:12.329597] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2546][SerScheQueue0][T1][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:01:12.329600] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000980) [2024-09-26 16:01:12.329603] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:01:12.329610] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=5][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:12.329613] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:01:12.329615] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:01:12.329618] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:01:12.329623] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:12.329651] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:01:12.329654] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:01:12.329659] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=4][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:01:12.329662] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:12.329665] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:12.329668] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:01:12.329790] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2546][SerScheQueue0][T0][YB42C0A8002D-0006230057984CE5-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:01:12.329797] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:01:12.329800] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2546][SerScheQueue0][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:01:12.329810] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=2] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:12.329822] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=11][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:01:12.330086] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:01:12.330093] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=7][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:12.339682] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:12.339691] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337672339680) [2024-09-26 16:01:12.339696] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337672239494, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:12.339704] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:12.339712] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:12.339715] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:12.340007] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:12.340020] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:12.340025] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:12.340028] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:12.340032] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:12.340034] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:12.340038] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:12.340040] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:12.340043] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:12.340045] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:12.340048] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:12.340053] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=5][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:12.340056] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:12.340058] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:12.340065] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=4][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:12.340068] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:12.340071] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:12.340076] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=4][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:12.340078] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:12.340081] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:12.340083] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:12.340088] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:12.340094] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:12.340095] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:12.340097] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:12.340102] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:12.340106] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:12.340108] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578E-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:12.340111] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:12.340114] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:12.340119] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:12.340121] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337672339940, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:12.340125] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:12.340127] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:12.340151] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:12.340154] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337672340153, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=445, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:12.340161] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:12.340163] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:12.340179] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223401576502, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:12.340184] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:12.340190] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:12.340194] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:12.340197] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:12.340202] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.340205] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.340207] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337672340167) [2024-09-26 16:01:12.351884] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.351892] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.366209] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.366219] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337672366206], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:12.366225] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337672366206], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:12.366229] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:12.366232] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=2] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:12.381936] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25] success to notify_need_writing_throttling True [2024-09-26 16:01:12.381944] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.393300] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=6] current monitor number(seq_=-1) [2024-09-26 16:01:12.394013] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=6] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:12.395655] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337672366206]}) [2024-09-26 16:01:12.411468] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:01:12.412060] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.412067] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.413704] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:12.413812] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Wash time detail, (compute_wash_size_time=59, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:12.414419] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1321194, clean_num=31457) [2024-09-26 16:01:12.436433] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=14][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337700364812, now=[mts=1727337671435903]) [2024-09-26 16:01:12.436445] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337700364812, now=[mts=1727337671435903], now0=[mts=1727337671435903], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:12.436452] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926902cea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337671435275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:12.436483] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=30][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f9269128038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926902cea0}) [2024-09-26 16:01:12.436491] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926902cea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337671435275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:12.436499] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c56d2378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926902cea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337671435275, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337670434817, use_das=false, nested_level=0, session={this:0x7f9269128038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926902cea0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:12.436518] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=18][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:12.436523] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:12.436526] WDIAG [SQL] open (ob_result_set.cpp:157) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:12.436529] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:12.436531] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:01:12.436535] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:12.436540] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4] will sleep(sleep_us=1000, remain_us=27998272, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4283, timeout_timestamp=1727337700434812) [2024-09-26 16:01:12.437568] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:12.437578] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:12.437581] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:12.437582] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:12.437588] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:12.437636] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=11][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=74) [2024-09-26 16:01:12.437639] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:12.439772] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:12.439788] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337672439769) [2024-09-26 16:01:12.439793] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337672339701, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:12.439812] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223401675954, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:12.439818] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:12.439826] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:12.439831] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:12.439834] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:12.439846] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.439849] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.439851] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337672439797) [2024-09-26 16:01:12.442131] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:12.442139] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.470869] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.470879] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.470885] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.470944] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=8] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.470951] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.470955] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.471793] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.471796] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.471798] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.471798] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.471804] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.471809] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.472203] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.472208] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.472355] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.472355] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.472361] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.472363] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.472358] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.472370] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=11][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.472880] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.472880] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.472887] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.472889] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.472884] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.472896] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=12][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.473120] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D4698-0-0] [lt=6][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:12.473484] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.473487] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.473489] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.473533] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.473540] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.473544] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.474682] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.474697] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=15][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.474700] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.474720] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.474724] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.474729] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.475370] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.475374] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.475375] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.475377] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.475378] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.475381] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.475907] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.475907] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.475910] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.475910] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.475912] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.475912] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:12.476558] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.476561] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.477101] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.477142] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.477631] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.477709] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.478209] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=11][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.478294] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.478796] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.478912] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.479376] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.479477] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.479953] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.480097] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.480477] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.480632] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.481192] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.481312] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.481758] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.481850] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.482305] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.482448] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.482882] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.483008] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.483445] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.483614] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.484001] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=12][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.484176] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=21][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.484647] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.484754] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.485257] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.485356] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.485771] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.485906] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.486310] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.486463] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.486932] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.487019] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.487537] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.487636] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.488070] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.488177] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.488648] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.488765] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.489230] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.489335] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.489923] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.490032] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.490484] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.490593] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.491039] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.491206] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.491584] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.491769] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.492222] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.492467] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.492759] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.492987] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.493281] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.493540] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.493845] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.494088] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.494407] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.494659] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.494985] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.495235] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.495579] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.495822] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.495830] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337672366206]}) [2024-09-26 16:01:12.496179] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.496414] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.496767] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.497060] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.497390] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.497668] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.497924] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.498296] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.498505] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.498908] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.499132] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.499516] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.499687] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.500040] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.500250] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.500643] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=29][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:12.502287] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:12.502309] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=20][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.524661] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1727337672524661, bytes: 14749792, bw: 0.002939 MB/s, add_ts: 1007570, add_bytes: 3105 [2024-09-26 16:01:12.527443] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185949-0-0] [lt=32][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:12.527467] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185949-0-0] [lt=23][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:12.527478] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185949-0-0] [lt=10][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:12.527484] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-0006230057185949-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:12.532425] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.532433] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.539827] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:12.539841] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337672539823) [2024-09-26 16:01:12.539847] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337672439797, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:12.539867] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223401775390, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:12.539873] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:12.539881] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:12.539885] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:12.539888] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:12.539903] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.539906] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.539911] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337672539850) [2024-09-26 16:01:12.551728] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6726, mem_used=12478464) [2024-09-26 16:01:12.562491] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.562499] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.568582] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=7] gc stale ls task succ [2024-09-26 16:01:12.592639] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.592646] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.595931] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337672366206]}) [2024-09-26 16:01:12.602209] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.602224] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.602230] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.602237] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=10] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.611529] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:01:12.614635] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:12.614743] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash time detail, (compute_wash_size_time=56, refresh_score_time=16, wash_time=2) [2024-09-26 16:01:12.615358] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1352651, clean_num=31457) [2024-09-26 16:01:12.622686] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.622693] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.633006] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=31] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:12.639964] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:12.639975] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337672639961) [2024-09-26 16:01:12.639980] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337672539850, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:12.639996] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223401875735, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:12.640001] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:12.640009] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:12.640014] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:12.640017] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:12.640023] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.640025] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.640027] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337672639983) [2024-09-26 16:01:12.652813] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.652836] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=21][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.667878] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:12.678829] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [2732][T1_Occam][T1][YB42C0A8002D-00062300566855FD-0-0] [lt=11] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:12.682986] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:12.682994] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.696145] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337672366206]}) [2024-09-26 16:01:12.705604] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:12.705628] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=22][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:12.705634] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:12.705638] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:12.705643] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:12.705645] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:12.705648] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:12.705651] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:12.705653] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:12.705655] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:12.705660] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:12.705663] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:12.705665] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:12.705667] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:12.705674] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:12.705677] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:12.705680] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:12.705683] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:12.705685] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server, ret=-5019) [2024-09-26 16:01:12.705688] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:12.705690] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:12.705696] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:12.705702] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:12.705705] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:12.705706] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:12.705713] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:12.705718] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2458][ServerGTimer][T1][YB42C0A8002D-0006230056A84DD9-0-0] [lt=4][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:12.705721] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2458][ServerGTimer][T0][YB42C0A8002D-0006230056A84DD9-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server"}, aret=-5019, ret=-5019) [2024-09-26 16:01:12.705724] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:12.705741] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:12.705743] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:12.705746] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337672705540, sql=SELECT *, time_to_usec(gmt_modified) AS last_hb_time FROM __all_server) [2024-09-26 16:01:12.705750] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:12.705784] WDIAG [SHARE] refresh (ob_all_server_tracer.cpp:568) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] fail to get servers_info(ret=-5019, ret="OB_TABLE_NOT_EXIST", GCTX.sql_proxy_=0x55c7486d2e88) [2024-09-26 16:01:12.705787] WDIAG [SHARE] runTimerTask (ob_all_server_tracer.cpp:626) [2458][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] fail to refresh all server map(ret=-5019) [2024-09-26 16:01:12.713093] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] success to notify_need_writing_throttling True [2024-09-26 16:01:12.713100] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.739725] INFO pn_ratelimit (group.c:551) [2692][IngressService][T0][Y0-0000000000000000-0-0] [lt=9] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2024-09-26 16:01:12.740056] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:12.740067] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337672740052) [2024-09-26 16:01:12.740073] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337672639983, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:12.740089] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223401976433, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:12.740095] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:12.740102] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:12.740107] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:12.740110] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:12.740115] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.740117] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.740119] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337672740077) [2024-09-26 16:01:12.743226] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:12.743234] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.773337] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:12.773348] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.796266] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337672366206]}) [2024-09-26 16:01:12.803914] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.803923] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.811631] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=8] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=2) [2024-09-26 16:01:12.815622] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:12.815729] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Wash time detail, (compute_wash_size_time=66, refresh_score_time=15, wash_time=1) [2024-09-26 16:01:12.816350] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1384108, clean_num=31457) [2024-09-26 16:01:12.820594] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.821040] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.821122] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.821489] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.821529] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.821542] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.821551] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.821558] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.821565] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.821572] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.821578] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:12.822675] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:12.834033] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=41] success to notify_need_writing_throttling True [2024-09-26 16:01:12.834043] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.840102] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:12.840113] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337672840099) [2024-09-26 16:01:12.840119] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337672740076, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:12.840139] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223402076177, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:12.840144] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:12.840152] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:12.840158] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:12.840160] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:12.840166] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.840169] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.840171] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337672840122) [2024-09-26 16:01:12.864096] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:12.864106] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.866601] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:12.866608] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337672866599], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:12.866615] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337672866599], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:12.894303] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:12.894323] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=18][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.896296] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337672866599]}) [2024-09-26 16:01:12.925647] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:12.925660] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.941527] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:12.941548] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337672941523) [2024-09-26 16:01:12.941554] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337672840122, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:12.941562] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:12.941589] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:12.941593] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:12.942133] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:12.942161] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:12.942165] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:12.942169] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:12.942172] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:12.942175] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:12.942178] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:12.942181] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:12.942183] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:12.942185] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:12.942187] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:12.942201] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=13][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:12.942204] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:12.942206] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:12.942213] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:12.942215] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:12.942218] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:12.942220] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:12.942223] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:12.942225] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:12.942228] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:12.942234] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:12.942240] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:12.942244] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:12.942245] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:12.942267] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:12.942272] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=4][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:12.942275] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-000623005788578F-0-0] [lt=3][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:12.942277] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:12.942280] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:12.942282] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:12.942284] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337672941932, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:12.942288] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:12.942304] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:12.942336] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:12.942339] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337672942338, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=771, wlock_time=28, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:12.942345] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:12.942349] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:12.942366] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223402178615, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:12.942388] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:12.942396] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:12.942400] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:12.942402] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:12.942408] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.942411] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:12.942412] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337672942352) [2024-09-26 16:01:12.955718] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:12.955730] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.979429] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1727337672979428, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007582, add_bytes: 0 [2024-09-26 16:01:12.980690] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:543) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=12][errcode=-4722] gts cache leader is invalid(ret=-4722, ret="OB_LS_LOCATION_LEADER_NOT_EXIST", leader="0.0.0.0:0", *this={tenant_id:1, gts_local_cache:{srr:[mts=0], gts:0, latest_srr:[mts=1727337672866599]}, server:"192.168.0.45:2882", gts_cache_leader:"0.0.0.0:0"}) [2024-09-26 16:01:12.980703] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:224) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=12][errcode=-4023] get gts leader fail(tmp_ret=-4722, tenant_id=1) [2024-09-26 16:01:12.985849] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24] success to notify_need_writing_throttling True [2024-09-26 16:01:12.985859] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:12.996143] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:12.996697] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337672866599]}) [2024-09-26 16:01:13.011726] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=1) [2024-09-26 16:01:13.016574] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:13.016690] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=54, refresh_score_time=15, wash_time=2) [2024-09-26 16:01:13.016801] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.016827] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.017331] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1415565, clean_num=31457) [2024-09-26 16:01:13.023415] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594A-0-0] [lt=42][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:13.023427] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594A-0-0] [lt=12][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:13.023436] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594A-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:13.023441] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594A-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:13.041605] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:13.041621] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337673041601) [2024-09-26 16:01:13.041646] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337672941557, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:13.041714] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=223402277700, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:13.041722] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:13.041730] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:13.041735] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:13.041738] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:13.041745] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.041766] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.041768] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337673041670) [2024-09-26 16:01:13.046894] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=28] success to notify_need_writing_throttling True [2024-09-26 16:01:13.046903] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.056021] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221500898, support_ssl=false) [2024-09-26 16:01:13.056041] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221500898, sess.client_addr_="127.0.0.1:36406") [2024-09-26 16:01:13.056045] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f9262604030, session_id:3221500898, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:13.056150] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:13.056161] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2691][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=5] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:13.056216] INFO load_privilege_info (obmp_connect.cpp:532) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FAA-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:13.056577] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FAA-0-0] [lt=5][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:13.056583] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FAA-0-0] [lt=5][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"{gnt}79V&Ym:yrOty%H{"}, ret=-4043) [2024-09-26 16:01:13.056590] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FAA-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:13.056593] WDIAG [SERVER] process (obmp_connect.cpp:340) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FAA-0-0] [lt=2][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:13.056603] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FAA-0-0] [lt=2] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:13.056637] INFO [SERVER] free_session (obmp_base.cpp:324) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FAA-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500898, proxy_sessid:0}) [2024-09-26 16:01:13.056642] INFO [SERVER] free_session (obmp_base.cpp:332) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FAA-0-0] [lt=5] mark session id unused(sessid=3221500898) [2024-09-26 16:01:13.056646] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FAA-0-0] [lt=2][errcode=0] server close connection(sessid=3221500898, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:13.056649] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FAA-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500898, proxy_sessid=0) [2024-09-26 16:01:13.056651] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FAA-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:13.056654] INFO [SERVER] process (obmp_connect.cpp:475) [2921][T1_MysqlQueueTh][T1][Y0-0006230056384FAA-0-0] [lt=2] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500898, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:13.056780] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f9262604030, session_id:3221500898, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337673056093, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:13.056806] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] kill and revert session(conn.sessid_=3221500898, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:13.056810] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=4] can close safely, do destroy(*s={this:0x7f9262604030, session_id:3221500898, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337673056093, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:13.056815] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2691][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500898, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:13.074776] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:13.074799] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=21][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, ret=-5019) [2024-09-26 16:01:13.074804] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=5][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_parameter, db_name=oceanbase) [2024-09-26 16:01:13.074808] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_parameter) [2024-09-26 16:01:13.074812] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:13.074814] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:13.074817] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_sys_parameter' doesn't exist [2024-09-26 16:01:13.074820] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:13.074822] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:13.074823] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:13.074826] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:13.074828] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:13.074830] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:13.074832] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:13.074838] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:13.074841] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:13.074844] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:13.074846] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:13.074849] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter, ret=-5019) [2024-09-26 16:01:13.074852] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:13.074855] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:13.074861] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:13.074867] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:13.074869] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:13.074871] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:13.074877] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:13.074881] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A2-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:13.074884] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter"}, aret=-5019, ret=-5019) [2024-09-26 16:01:13.074887] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:13.074890] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:13.074892] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:13.074898] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A2-0-0] [lt=5][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337673074706, sql=select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter) [2024-09-26 16:01:13.074902] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A2-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:13.074904] WDIAG [SHARE] update_local (ob_config_manager.cpp:354) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A2-0-0] [lt=1][errcode=-5019] read config from __all_sys_parameter failed(sqlstr="select config_version, zone, svr_type, svr_ip, svr_port, name, data_type, value, info, section, scope, source, edit_level from __all_sys_parameter", ret=-5019) [2024-09-26 16:01:13.074925] WDIAG [SHARE] update_local (ob_config_manager.cpp:391) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A2-0-0] [lt=2][errcode=-5019] Read system config from inner table error(ret=-5019) [2024-09-26 16:01:13.074929] WDIAG [SHARE] runTimerTask (ob_config_manager.cpp:490) [2464][ConfigMgr][T0][YB42C0A8002D-0006230057C858A2-0-0] [lt=4][errcode=-5019] Update local config failed(ret=-5019) [2024-09-26 16:01:13.077024] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.077032] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.079210] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=9] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:13.079519] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:13.079529] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=9][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:13.079533] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:13.079536] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:13.079539] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:13.079549] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=10][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:13.079552] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:13.079554] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:13.079556] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:13.079558] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:13.079559] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:13.079562] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=3][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:13.079564] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:13.079566] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:13.079571] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:13.079573] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:13.079584] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E82-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:13.092893] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:13.097282] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337672866599]}) [2024-09-26 16:01:13.107233] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.107242] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.137302] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:13.137311] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.141876] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:13.141886] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:13.141893] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337673141873) [2024-09-26 16:01:13.141898] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337673041670, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:13.141916] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223402378353, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:13.141921] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:13.141930] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:13.141936] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:13.141939] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:13.141945] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.141947] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.141949] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337673141901) [2024-09-26 16:01:13.168217] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.168243] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.192514] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:256) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=10][errcode=-4018] try free send task failed(ret=-4018) [2024-09-26 16:01:13.196587] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 10719.00) [2024-09-26 16:01:13.196598] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:13.196612] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:13.196616] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=4] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:13.197360] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337672866599]}) [2024-09-26 16:01:13.198342] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:13.198350] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.211778] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=0) [2024-09-26 16:01:13.214514] WDIAG [SERVER] check_and_update_server_id_ (ob_heartbeat.cpp:188) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4002] invalid server_id(ret=-4002, ret="OB_INVALID_ARGUMENT", server_id=18446744073709551615) [2024-09-26 16:01:13.214528] INFO [SERVER] try_reload_schema (ob_server_schema_updater.cpp:440) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=7] schedule fetch new schema task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:13.214533] INFO [SERVER] do_heartbeat_event (ob_heartbeat.cpp:268) [3026][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=5] try reload schema success(schema_version=1, refresh_schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}, schema_ret=0) [2024-09-26 16:01:13.217569] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:13.217747] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=62] Wash time detail, (compute_wash_size_time=100, refresh_score_time=40, wash_time=2) [2024-09-26 16:01:13.218449] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1447022, clean_num=31457) [2024-09-26 16:01:13.228396] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:13.228403] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.242426] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:13.242437] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337673242423) [2024-09-26 16:01:13.242442] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337673141901, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:13.242461] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223402478183, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:13.242467] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:13.242475] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:13.242480] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:13.242482] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:13.242487] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.242489] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.242491] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337673242446) [2024-09-26 16:01:13.258595] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:13.258604] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.268389] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:13.268401] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=11][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2024-09-26 16:01:13.268405] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:13.268408] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2024-09-26 16:01:13.268412] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:13.268414] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:13.268418] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2024-09-26 16:01:13.268420] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:13.268423] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:13.268424] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:13.268428] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=4][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:13.268430] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:13.268453] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B1-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:13.282538] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=9][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:53, tid:2919}]) [2024-09-26 16:01:13.289379] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:13.289388] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.292562] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:262) [2941][T1_ArcSender][T1][YB42C0A8002D-0006230055F84B24-0-0] [lt=11] ObArchiveSender is running(thread_index=0) [2024-09-26 16:01:13.297782] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337672866599]}) [2024-09-26 16:01:13.297799] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8DA26-0-0] [lt=8][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337673297775}) [2024-09-26 16:01:13.319470] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:13.319479] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.319490] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:13.320172] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:138) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DDA-0-0] [lt=2][errcode=-4002] invalid argument, empty server list(ret=-4002) [2024-09-26 16:01:13.320180] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:380) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DDA-0-0] [lt=7][errcode=-4002] refresh sever list failed(ret=-4002) [2024-09-26 16:01:13.320182] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:255) [2461][ServerTracerTim][T0][YB42C0A8002D-0006230056984DDA-0-0] [lt=2][errcode=-4002] refresh alive server list failed(ret=-4002) [2024-09-26 16:01:13.330679] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337674259829, now=[mts=1727337672330085]) [2024-09-26 16:01:13.330722] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=42][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337674259829, now=[mts=1727337672330085], now0=[mts=1727337672330085], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:13.330730] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=7][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926916a8c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337672329279, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:13.330748] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926916a8c0}) [2024-09-26 16:01:13.330756] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926916a8c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337672329279, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:13.330764] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926916a8c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337672329279, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337672330056, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926916a8c0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:13.330781] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:13.330786] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:13.330789] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:13.330793] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:13.330795] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:13.330799] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:13.330804] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:103) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4] already timeout, do not need sleep(sleep_us=0, remain_us=999025, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=0, timeout_timestamp=1727337674329829) [2024-09-26 16:01:13.330808] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:13.330815] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:13.330817] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:13.330818] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=1][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:13.330824] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:13.330861] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:01:13.330864] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:13.342639] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:13.342650] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337673342635) [2024-09-26 16:01:13.342655] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337673242446, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:13.342672] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223402578845, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:13.342678] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:13.342684] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:13.342691] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:13.342694] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:13.342699] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.342701] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.342703] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337673342658) [2024-09-26 16:01:13.349560] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:13.349570] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.367585] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.367597] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337673367583], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:13.367605] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337673367583], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:13.367608] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:13.367613] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=4] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:13.379694] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:13.379704] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.393475] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:331) [3687][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=7] current monitor number(seq_=-1) [2024-09-26 16:01:13.394093] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:13.397830] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337673367583]}) [2024-09-26 16:01:13.409828] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:13.409837] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.412332] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:696) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=13] Cache replace map node details(ret=0, replace_node_count=0, replace_time=0, replace_start_pos=990864, replace_num=15728) [2024-09-26 16:01:13.412342] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=10) [2024-09-26 16:01:13.418836] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=53] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:13.419014] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=138, refresh_score_time=19, wash_time=1) [2024-09-26 16:01:13.420062] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1478479, clean_num=31457) [2024-09-26 16:01:13.438207] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=11][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337700364812, now=[mts=1727337672437635]) [2024-09-26 16:01:13.438240] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=32][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337700364812, now=[mts=1727337672437635], now0=[mts=1727337672437635], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:13.438247] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926902cea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337672437408, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:13.438264] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=16][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f9269128038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926902cea0}) [2024-09-26 16:01:13.438283] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=18][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926902cea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337672437408, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:13.438291] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c56d2378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926902cea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337672437408, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337670434817, use_das=false, nested_level=0, session={this:0x7f9269128038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926902cea0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:13.438308] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=16][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:13.438313] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:13.438316] WDIAG [SQL] open (ob_result_set.cpp:157) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:13.438318] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:13.438321] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:01:13.438324] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=2, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:13.438329] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3] will sleep(sleep_us=2000, remain_us=26996484, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4283, timeout_timestamp=1727337700434812) [2024-09-26 16:01:13.438517] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=14] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:01:13.438529] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=7] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:13.438536] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:13.438539] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:13.438542] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:13.438545] INFO [SHARE.SCHEMA] get_database_schema (ob_schema_getter_guard.cpp:2217) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=2] database not exist(tenant_id=1, database_name=oceanbase) [2024-09-26 16:01:13.438547] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:13.438549] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:13.438553] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:13.438555] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:13.438573] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=18][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:13.438575] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:13.438577] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:13.438580] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:13.438582] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:13.438584] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=2][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:13.438590] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:13.438593] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:13.438596] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:13.438598] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:13.438600] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=select * from oceanbase.__all_server, ret=-5019) [2024-09-26 16:01:13.438605] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=2][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:false, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:13.438609] WDIAG [SERVER] do_process (obmp_query.cpp:771) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=4][errcode=-5019] run stmt_query failed, check if need retry(ret=-5019, cli_ret=-5019, retry_ctrl_.need_retry()=0, sql=select * from oceanbase.__all_server) [2024-09-26 16:01:13.438615] WDIAG [SERVER] do_process (obmp_query.cpp:903) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=3][errcode=-5019] query failed(ret=-5019, session={this:0x7f92a230bdf8, id:3221487638, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"", user:"root@%", consistency_level:3, session_state:2, autocommit:true, tx:null}, sql=select * from oceanbase.__all_server, retry_ctrl_.need_retry()=0) [2024-09-26 16:01:13.438624] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056E84F58-0-0] [lt=5] sending error packet(ob_error=-5019, client error=1146, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0x4e364a2 0x4e2ab67 0x4e22201 0x4e1fb17 0x4e0f175 0xa7ab0b4 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:13.439922] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.439931] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.440385] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:13.440397] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=8][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:13.440399] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:13.440401] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:13.440407] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:13.440463] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=74) [2024-09-26 16:01:13.440469] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:13.442680] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:140) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] ====== tenant freeze timer task ====== [2024-09-26 16:01:13.442676] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:13.442690] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1606) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=48234496, ctx.total_memstore_used_=759169024, ctx.total_memstore_hold_=759169024, memstore_freeze_trigger_limit_=214748360, tenant_id=1) [2024-09-26 16:01:13.442702] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337673442672) [2024-09-26 16:01:13.442709] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337673342658, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:13.442714] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:13.442720] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1615) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4] [TenantFreezer] oldest frozen memtable(list="{ObITable:{this:0x7f92a1a70080, key:{tablet_id:{id:200489}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170043069079001, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199526729}, this:0x7f92a1a70080, timestamp:1727334199526729, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1725120002010696, max_column_cnt:17, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:1, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170043069079001, v:0}, rec_scn:{val:1726163995353415001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401494, mt_stat_.ready_for_flush_time:1727334201452884, mt_stat_.create_flush_dag_time:1727337670636259, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a70850, key:{tablet_id:{id:200240}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726163995200406000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199527498}, this:0x7f92a1a70850, timestamp:1727334199527498, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:18, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:2, clock:0}, host:0x7f92ba380030, arena_handle:{allocated:98566144}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726163995323287001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401489, mt_stat_.ready_for_flush_time:1727334201452830, mt_stat_.create_flush_dag_time:1727337670636210, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_stat_.last_print_time:0, ls_id:{id:1}, transfer_freeze_flag:false, recommend_snapshot_version:{val:18446744073709551615, v:3}},{ObITable:{this:0x7f92a1a71020, key:{tablet_id:{id:1152921504606847222}, column_group_idx:0, table_type:"MEMTABLE", scn_range:{start_scn:{val:1726164781225658000, v:0}, end_scn:{val:1726170015675149000, v:0}}}, ref_cnt:2, upper_trans_version:9223372036854775807, timestamp:1727334199757623}, this:0x7f92a1a71020, timestamp:1727334199757623, state:0, freeze_clock:0, max_schema_version:0, max_data_schema_version:1711680543204904, max_column_cnt:2, write_ref_cnt:0, local_allocator:{ListHandle:{freeze_stat:2, id:3, clock:25165824}, host:0x7f92ba380030, arena_handle:{allocated:106954752}}, unsubmitted_cnt:0, unsynced_cnt:0, logging_blocked:false, unset_active_memtable_logging_blocked:true, resolve_active_memtable_left_boundary:true, contain_hotspot_row:false, max_end_scn:{val:1726170015675149000, v:0}, rec_scn:{val:1726164783141994001, v:0}, snapshot_version:{val:1726168982778454000, v:0}, migration_clog_checkpoint_scn:{val:0, v:0}, is_tablet_freeze:false, is_force_freeze:false, contain_hotspot_row:false, read_barrier:false, is_flushed:false, freeze_state:1, allow_freeze:true, mt_stat_.frozen_time:1727334201401495, mt_stat_.ready_for_flush_time:1727334201452904, mt_stat_.create_flush_dag_time:1727337670636278, mt_stat_.release_time:0, mt_stat_.push_table_into_gc_queue_time:0, mt_s") [2024-09-26 16:01:13.442722] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:13.442725] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:13.442743] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:560) [2839][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=22, Tenant Active TxData Memory(MB)=76, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=128, Total TxDataTable Memory Limit(MB)=204) [2024-09-26 16:01:13.443046] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:13.443053] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:13.443056] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:13.443059] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:13.443061] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:13.443063] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:13.443066] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:13.443068] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:13.443069] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:13.443071] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:13.443072] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:13.443074] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:13.443076] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:13.443077] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:13.443081] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:13.443083] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:13.443085] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:13.443087] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:13.443088] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:13.443090] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:13.443092] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:13.443098] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:13.443103] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:13.443105] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:13.443106] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:13.443110] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:13.443113] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:13.443126] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885790-0-0] [lt=13][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:13.443130] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:13.443132] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:13.443134] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:13.443137] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337673442981, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:13.443140] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:13.443143] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:13.443169] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:13.443171] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337673443170, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=451, wlock_time=8, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:13.443177] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:13.443180] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:13.443194] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223402679231, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:13.443198] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:13.443204] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:13.443208] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:13.443210] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:13.443215] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.443217] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.443218] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337673443183) [2024-09-26 16:01:13.443244] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:80) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E9-0-0] [lt=4] start tx data table self freeze task in rpc handle thread(arg_=freeze_type:3) [2024-09-26 16:01:13.443249] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:788) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E9-0-0] [lt=3] start tx data table self freeze task(get_ls_id()={id:1}) [2024-09-26 16:01:13.443253] INFO [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:223) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E9-0-0] [lt=3] start freeze tx data memtable(ls_id_={id:1}) [2024-09-26 16:01:13.443256] INFO [STORAGE] freeze_ (ob_tx_data_memtable_mgr.cpp:259) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E9-0-0] [lt=2] There is a freezed memetable existed. Try freeze after flushing it.(ret=-4023, ret="OB_EAGAIN", get_memtable_count_()=2) [2024-09-26 16:01:13.443259] WDIAG [STORAGE] freeze (ob_tx_data_memtable_mgr.cpp:237) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E9-0-0] [lt=2][errcode=-4023] freeze tx data memtable fail.(ret=-4023, ret="OB_EAGAIN") [2024-09-26 16:01:13.443261] WDIAG [STORAGE] flush (ob_tx_data_memtable_mgr.cpp:511) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E9-0-0] [lt=2][errcode=-4023] freeze failed(ret=-4023, ret="OB_EAGAIN", this=0x7f92a6c041b0) [2024-09-26 16:01:13.443263] WDIAG [STORAGE] self_freeze_task (ob_tx_data_table.cpp:792) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E9-0-0] [lt=2][errcode=-4023] self freeze of tx data memtable failed.(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}, memtable_mgr_={ObIMemtableMgr:{Memtables:this:0x7f92a6c041b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f92a6c09cd0, table_type:1, memtable_head:0, memtable_tail:2, t3m:0x7f92cfbda030, tables:[0x7f92a2294080, 0x7f92a2294b90, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f92a6c0b1d0, ls_tablet_svr:0x7f92a6c04190, slice_allocator:0x7f92a6c0b210}) [2024-09-26 16:01:13.443271] INFO [STORAGE] self_freeze_task (ob_tx_data_table.cpp:795) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E9-0-0] [lt=8] finish tx data table self freeze task(ret=-4023, ret="OB_EAGAIN", get_ls_id()={id:1}) [2024-09-26 16:01:13.443273] WDIAG [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:109) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E9-0-0] [lt=2][errcode=-4023] freeze tx data table failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:13.443275] INFO [STORAGE] do_tx_data_table_freeze_ (ob_tenant_freezer_rpc.cpp:122) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E9-0-0] [lt=1] finish self freeze task in rpc handle thread(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:13.443277] WDIAG [STORAGE] process (ob_tenant_freezer_rpc.cpp:59) [3001][T1_L0_G0][T1][YB42C0A8002D-0006230056F851E9-0-0] [lt=1][errcode=-4023] do tx data table freeze failed.(ret=-4023, ret="OB_EAGAIN", arg_=freeze_type:3) [2024-09-26 16:01:13.443305] INFO [STORAGE] rpc_callback (ob_tenant_freezer.cpp:1356) [2533][pnio1][T0][YB42C0A8002D-0006230056F851E9-0-0] [lt=7] [TenantFreezer] call back of tenant freezer request [2024-09-26 16:01:13.470034] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.470044] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.470977] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.470987] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.470994] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=6][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.470999] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.471002] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.471006] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.471524] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.471530] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.471533] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.471619] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.471621] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.471623] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.472314] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.472321] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.472325] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.472329] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=82] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.472331] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.472333] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.473022] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.473027] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.473030] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.473033] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.473035] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.473036] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.473620] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.473625] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.473628] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.473631] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.473632] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.473634] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.474216] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.474222] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.474224] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.474227] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.474229] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.474230] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.474791] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.474794] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.474796] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.474799] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.474801] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.474802] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.475389] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.475395] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.475397] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.475401] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.475403] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.475405] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:13.476002] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.476006] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.476282] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2569][LSMetaTblUp0][T0][YB42C0A8002D-00062300559D46FB-0-0] [lt=6][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:13.476534] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.476538] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.477080] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.477084] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.477796] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.477801] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.478499] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.478504] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.479053] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.479058] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.479594] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.479598] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.480309] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.480315] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.480857] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.480864] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.481494] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.481502] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.482024] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.482186] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=24][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.482568] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.482774] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.483139] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.483355] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.483707] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.483968] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.484254] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.484517] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.484782] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.485060] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.485422] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.485666] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.486019] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.486267] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.486549] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.486852] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.487124] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.487447] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.487762] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.488047] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.488438] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.488642] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.489051] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.489241] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.489590] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.489812] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.490219] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.490385] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=31][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.490756] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.490959] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.491350] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.491474] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.491912] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.492049] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.492489] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.492634] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.493100] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.493255] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.493715] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.493786] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.494355] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.494365] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=22][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.494928] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.494936] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.495469] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.495474] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.496059] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.496070] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.496632] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.496654] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.497239] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=20][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.497245] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.497875] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.497884] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337673367583]}) [2024-09-26 16:01:13.497922] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.498527] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=27][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.498534] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.499111] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.499119] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.499711] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.499715] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.500155] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:13.500165] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.500345] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.500350] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:13.522612] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594B-0-0] [lt=51][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:13.522624] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594B-0-0] [lt=11][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:13.522634] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594B-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:13.522639] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594B-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:13.530205] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.530216] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.532332] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=11] PNIO [ratelimit] time: 1727337673532331, bytes: 14755340, bw: 0.005251 MB/s, add_ts: 1007670, add_bytes: 5548 [2024-09-26 16:01:13.542740] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:13.542753] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337673542736) [2024-09-26 16:01:13.542758] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337673442712, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:13.542778] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223402778442, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:13.542784] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:13.542792] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:13.542796] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:13.542799] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:13.542805] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.542807] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.542809] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337673542762) [2024-09-26 16:01:13.551839] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=1, size_used=6735, mem_used=12478464) [2024-09-26 16:01:13.560327] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:13.560335] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.568669] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=6] gc stale ls task succ [2024-09-26 16:01:13.590531] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:13.590541] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.591735] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:218) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-09-26 16:01:13.591744] INFO [COMMON] handle (memory_dump.cpp:498) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2024-09-26 16:01:13.591770] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3] update_check_range(min_check_version=344, max_check_version=345, global_version=346) [2024-09-26 16:01:13.594204] INFO handle (memory_dump.cpp:626) [2512][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=3] statistics: tenant_cnt: 3, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 80 198 0 1 2 362 1 0 1 4 1 1 0 1 5 1 5 0 1 7 1 2 0 1 8 60 1 0 1 12 4 2 0 1 13 2 1 0 1 14 1 1 0 1 18 54 3 0 500 0 52 216 0 500 7 2 3 0 500 8 42 1 0 500 9 2 1 0 500 10 6 2 0 500 16 1 1 0 500 17 6 7 0 500 22 3 49 0 500 23 23 10 0 508 0 2 8 0 508 8 3 1 0 cost_time: 2437 [2024-09-26 16:01:13.598213] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337673367583]}) [2024-09-26 16:01:13.601614] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.601626] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.601632] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.601638] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.612399] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=9) [2024-09-26 16:01:13.620265] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=8] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:13.620433] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=61] Wash time detail, (compute_wash_size_time=101, refresh_score_time=20, wash_time=2) [2024-09-26 16:01:13.620632] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.620638] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.621181] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1509936, clean_num=31457) [2024-09-26 16:01:13.626598] INFO [PALF] log_loop_ (log_loop_thread.cpp:155) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] LogLoopThread round_cost_time(us)(round_cost_time=9) [2024-09-26 16:01:13.637401] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=22] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:13.637415] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2024-09-26 16:01:13.637419] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2024-09-26 16:01:13.637422] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2024-09-26 16:01:13.637425] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:13.637428] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=3][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:13.637443] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2024-09-26 16:01:13.637446] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:13.637448] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:13.637450] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:13.637452] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:13.637455] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:13.637457] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=1][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:13.637459] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:13.637465] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:13.637468] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=3][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:13.637471] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:13.637494] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:13.637498] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=3][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, ret=-5019) [2024-09-26 16:01:13.637501] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:13.637503] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:13.637509] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=5][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:13.637515] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:13.637518] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:13.637519] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:13.637525] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:13.637529] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:13.637532] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2736][T1_Occam][T1][YB42C0A8002D-0006230056C84DDA-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882"}, aret=-5019, ret=-5019) [2024-09-26 16:01:13.637536] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:01:13.637539] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:13.637542] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:13.637544] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8340, start=1727337673637319, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:01:13.637548] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:13.637551] WDIAG get_my_sql_result_ (ob_table_access_helper.h:435) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f92b4056738, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='192.168.0.45' and svr_port=2882, columns_str="zone") [2024-09-26 16:01:13.637556] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:332) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='192.168.0.45' and svr_port=2882) [2024-09-26 16:01:13.637585] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:634) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f92b4056738, where_condition="where svr_ip='192.168.0.45' and svr_port=2882", zone_name_holder=) [2024-09-26 16:01:13.637590] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:567) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:01:13.637593] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:576) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2024-09-26 16:01:13.637595] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:144) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:13.637600] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:13.642768] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223402878315, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:13.642778] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:13.642786] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:13.642792] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:13.642795] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:13.642801] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.642803] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.642805] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337673642753) [2024-09-26 16:01:13.650685] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:13.650692] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.668016] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=40] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:13.680932] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:13.680950] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.698715] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337673367583]}) [2024-09-26 16:01:13.700505] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [2899][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=19] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:1536, free_obj_hold_cnt:0, allocator used:6107136, allocator total:6279168}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6279168, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:89, free_obj_hold_cnt:0, allocator used:176576, allocator total:196224}, pending_cnt=0, wait_gc_count=0, tablet count=1536) [2024-09-26 16:01:13.710991] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.711000] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.741149] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:13.741161] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.742810] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:13.742821] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337673742807) [2024-09-26 16:01:13.742827] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337673542761, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:13.742845] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223402978644, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:13.742850] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:13.742858] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:13.742863] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:13.742865] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:13.742871] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.742873] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.742875] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337673742830) [2024-09-26 16:01:13.771246] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.771254] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.799172] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337673367583]}) [2024-09-26 16:01:13.801321] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.801331] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.812710] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=8) [2024-09-26 16:01:13.821125] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.821140] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=6] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.821431] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=73] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:13.821556] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash time detail, (compute_wash_size_time=72, refresh_score_time=16, wash_time=2) [2024-09-26 16:01:13.822334] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1541393, clean_num=31457) [2024-09-26 16:01:13.822351] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=3] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.822359] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=31] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.822480] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.822490] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.822489] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.822496] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.822500] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.822505] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.822511] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:13.826759] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:13.831460] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.831467] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.842957] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:13.842970] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337673842953) [2024-09-26 16:01:13.842976] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337673742830, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:13.842994] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223403078608, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:13.843000] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:13.843008] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:13.843013] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:13.843016] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:13.843022] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.843025] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.843027] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337673842979) [2024-09-26 16:01:13.861689] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:13.861698] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.867960] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:13.867969] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337673867958], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:13.867976] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337673867958], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:13.891844] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:13.891855] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.899631] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337673867958]}) [2024-09-26 16:01:13.922014] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:13.922022] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.943012] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:13.943032] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337673943008) [2024-09-26 16:01:13.943038] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337673842979, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:13.943059] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223403179256, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:13.943066] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:13.943093] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:13.943099] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:13.943102] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:13.943109] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.943138] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:13.943142] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337673943041) [2024-09-26 16:01:13.952117] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:13.952126] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.966436] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2024-09-26 16:01:13.966460] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=22][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2024-09-26 16:01:13.966465] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2024-09-26 16:01:13.966468] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2024-09-26 16:01:13.966472] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:13.966474] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:13.966477] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2024-09-26 16:01:13.966480] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:13.966482] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:13.966484] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:13.966486] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:13.966488] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:13.966491] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:13.966492] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:13.966499] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=3][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:13.966501] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:13.966504] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:13.966506] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:13.966508] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:01:13.966511] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:13.966513] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:13.966519] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:13.966524] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=4][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:13.966527] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:13.966528] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:13.966534] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:13.966537] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=3][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:13.966540] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2024-09-26 16:01:13.966543] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:01:13.966545] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:13.966547] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:13.966549] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337673966348, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:01:13.966552] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:13.966555] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1111) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2024-09-26 16:01:13.966580] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:801) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2024-09-26 16:01:13.966582] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:67) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:13.966584] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:436) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:13.966586] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:95) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:13.966588] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:109) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:13.966788] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2024-09-26 16:01:13.966793] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=4][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2024-09-26 16:01:13.966795] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2024-09-26 16:01:13.966797] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2024-09-26 16:01:13.966799] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:13.966800] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:13.966802] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_tenant' doesn't exist [2024-09-26 16:01:13.966804] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:13.966805] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:13.966807] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:13.966808] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:13.966809] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:13.966811] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:13.966812] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:13.966814] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:13.966816] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:13.966817] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:13.966819] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:13.966820] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:01:13.966822] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:13.966823] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:13.966826] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:13.966829] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:13.966830] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:13.966831] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:13.966834] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:13.966836] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2709][OmtNodeBalancer][T1][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:13.966838] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, aret=-5019, ret=-5019) [2024-09-26 16:01:13.966839] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:01:13.966841] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:13.966843] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:13.966844] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337673966766, sql=SELECT tenant_id FROM __all_tenant) [2024-09-26 16:01:13.966846] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:13.966848] WDIAG [SHARE] read_tenants (ob_unit_table_operator.cpp:1174) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] execute sql failed(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:01:13.966857] WDIAG [SHARE] get_tenants (ob_unit_table_operator.cpp:108) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=1][errcode=-5019] fail read tenants(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2024-09-26 16:01:13.966859] WDIAG [SHARE] get_tenants (ob_unit_getter.cpp:198) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] ut_operator get_resource_pools failed(ret=-5019) [2024-09-26 16:01:13.966861] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:125) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2][errcode=-5019] get cluster tenants fail(ret=-5019) [2024-09-26 16:01:13.966871] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:325) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=1] cal_all_part_disk_default_percentage succ(data_dir="/root/observer/store/sstable", clog_dir="/root/observer/store/clog", shared_mode=true, data_disk_total_size=99765694464, data_disk_default_percentage=60, clog_disk_total_size=99765694464, clog_disk_default_percentage=30) [2024-09-26 16:01:13.966875] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:361) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=4] decide disk size finished(suggested_disk_size=15032385536, suggested_disk_percentage=0, default_disk_percentage=30, total_space=99765694464, disk_size=15032385536) [2024-09-26 16:01:13.966878] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2] get_log_disk_info_in_config(suggested_data_disk_size=2147483648, suggested_clog_disk_size=15032385536, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=15032385536, log_disk_percentage=0, total_log_disk_size=99765694464) [2024-09-26 16:01:13.966881] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:785) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=2] try_resize success(ret=0, log_disk_size=15032385536, total_log_disk_size=99765694464, this={dir::"/root/observer/store/clog/log_pool", dir_fd:15, meta_fd:16, log_pool_meta:{curr_total_size:15032385536, next_total_size:15032385536, status:0}, min_block_id:9388, max_block_id:9581, min_log_disk_size_for_all_tenants_:2147483648, is_inited:true}) [2024-09-26 16:01:13.966885] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:133) [2709][OmtNodeBalancer][T0][YB42C0A8002D-0006230055384DDB-0-0] [lt=4] refresh tenant config(tenants=[], ret=-5019) [2024-09-26 16:01:13.982174] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:13.982181] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.982194] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] BlockGCTimerTask success(ret=0, cost_time_us=21, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:13.987091] INFO eloop_run (eloop.c:143) [2535][pnio2][T0][Y0-0000000000000000-0-0] [lt=12] PNIO [ratelimit] time: 1727337673987090, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007662, add_bytes: 0 [2024-09-26 16:01:13.996988] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:264) [2949][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1}) [2024-09-26 16:01:13.999641] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337673867958]}) [2024-09-26 16:01:14.012264] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6] success to notify_need_writing_throttling True [2024-09-26 16:01:14.012271] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.012818] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=10] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=7) [2024-09-26 16:01:14.022554] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2254405632, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:14.022736] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash time detail, (compute_wash_size_time=61, refresh_score_time=16, wash_time=2) [2024-09-26 16:01:14.022746] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=1572850, clean_num=31457) [2024-09-26 16:01:14.027311] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FAA-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:14.027324] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FAA-0-0] [lt=12][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2024-09-26 16:01:14.027328] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FAA-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2024-09-26 16:01:14.027332] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FAA-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2024-09-26 16:01:14.027335] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FAA-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:14.027337] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FAA-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:14.027340] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FAA-0-0] [lt=2][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2024-09-26 16:01:14.027343] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FAA-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:14.027347] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FAA-0-0] [lt=4][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:14.027349] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FAA-0-0] [lt=2][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:14.027351] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:3268) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FAA-0-0] [lt=2][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:14.027384] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [3024][BlackListServic][T1][YB42C0A8002D-0006230056284FAA-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:14.027394] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:398) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=0] start to print blacklist info [2024-09-26 16:01:14.027418] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:194) [3024][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=3] ls blacklist refresh finish(cost_time=511) [2024-09-26 16:01:14.028901] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594C-0-0] [lt=38][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:14.028912] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594C-0-0] [lt=10][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:14.028921] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594C-0-0] [lt=8][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:14.028926] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594C-0-0] [lt=4][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:14.042366] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:14.042374] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.043051] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:14.043061] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337674043048) [2024-09-26 16:01:14.043067] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337673943041, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:14.043072] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:14.043081] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:14.043083] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:14.043384] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885791-0-0] [lt=3] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:14.043395] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885791-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:14.043428] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885791-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:14.043454] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337674043454, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=377, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:14.043463] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:14.043478] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223403279897, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:14.043484] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:14.043490] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:14.043495] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:14.043498] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:14.043503] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.043506] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.043507] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337674043467) [2024-09-26 16:01:14.058098] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] new mysql sessid created(conn.sessid_=3221500903, support_ssl=false) [2024-09-26 16:01:14.058117] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221500903, sess.client_addr_="127.0.0.1:36408") [2024-09-26 16:01:14.058121] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1024) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] accept one succ(*s={this:0x7f92be3fb1f0, session_id:3221500903, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:140}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:14.058160] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:79) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2024-09-26 16:01:14.058169] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:275) [2690][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=3] succeed to dispatch to tenant mysql queue(tenant_id=1) [2024-09-26 16:01:14.058201] INFO load_privilege_info (obmp_connect.cpp:532) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FAA-0-0] [lt=6] no tenant name set, use default tenant name(tenant_name=sys) [2024-09-26 16:01:14.058415] WDIAG [SHARE.SCHEMA] check_user_access (ob_schema_getter_guard.cpp:2605) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FAA-0-0] [lt=3][errcode=-4043] password error(ret=-4043, ret="OB_PASSWORD_WRONG", login_info.passwd_.length()=20, user_info->get_passwd_str().length()=0) [2024-09-26 16:01:14.058421] WDIAG [SERVER] load_privilege_info (obmp_connect.cpp:665) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FAA-0-0] [lt=5][errcode=-4043] User access denied(login_info={tenant_name:"sys", user_name:"root", client_ip:"127.0.0.1", db:"", scramble_str:"n"Xyvds5NhCRR&bAZ$@U"}, ret=-4043) [2024-09-26 16:01:14.058428] WDIAG [SERVER] verify_identify (obmp_connect.cpp:1926) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FAA-0-0] [lt=5][errcode=-4043] load privilege info fail(pre_ret=-4043, ret=-4043, GCTX.status_=1) [2024-09-26 16:01:14.058432] WDIAG [SERVER] process (obmp_connect.cpp:340) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FAA-0-0] [lt=3][errcode=-4043] fail to verify_identify(ret=-4043) [2024-09-26 16:01:14.058439] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:366) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FAA-0-0] [lt=1] sending error packet(ob_error=-4043, client error=1045, extra_err_info=NULL, lbt()="0x121f1f9c 0xa376430 0xa324d42 0xa3564e6 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:14.058475] INFO [SERVER] free_session (obmp_base.cpp:324) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FAA-0-0] [lt=3] free session successfully(ctx={has_inc_active_num:false, tenant_id:1, sessid:3221500903, proxy_sessid:0}) [2024-09-26 16:01:14.058480] INFO [SERVER] free_session (obmp_base.cpp:332) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FAA-0-0] [lt=4] mark session id unused(sessid=3221500903) [2024-09-26 16:01:14.058483] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:830) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FAA-0-0] [lt=1][errcode=0] server close connection(sessid=3221500903, proxy_sessid=0, stack="0x121f1f9c 0xa379bb0 0xa344a5b 0xa355d37 0x4e22201 0x13856142 0x1385739a 0x13857919 0x12d223f1 0x12d1e6ac 0x7f92d3a081cf 0x7f92d3639d83") [2024-09-26 16:01:14.058486] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:592) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FAA-0-0] [lt=3][errcode=-4018] get session fail(ret=-4018, sessid=3221500903, proxy_sessid=0) [2024-09-26 16:01:14.058489] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:834) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FAA-0-0] [lt=2][errcode=-4016] session is null [2024-09-26 16:01:14.058491] INFO [SERVER] process (obmp_connect.cpp:475) [2922][T1_MysqlQueueTh][T1][Y0-0006230056584FAA-0-0] [lt=1] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=127.0.0.1, tenant_name=sys, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221500903, proxy_sessid=0, sess_create_time=0, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=762501, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-4043, ret=0, conn->client_type_=3, conn->client_version_=0) [2024-09-26 16:01:14.058526] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:824) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4015] close sql sock by user req(*s={this:0x7f92be3fb1f0, session_id:3221500903, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337674056735, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2024-09-26 16:01:14.058535] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:268) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] kill and revert session(conn.sessid_=3221500903, proxy_sessid=0, server_id=1, ret=0) [2024-09-26 16:01:14.058538] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:920) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] can close safely, do destroy(*s={this:0x7f92be3fb1f0, session_id:3221500903, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:140}, err:5, last_decode_time:1727337674056735, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:84, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2024-09-26 16:01:14.058542] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [2690][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=3] connection close(sessid=3221500903, proxy_sessid=0, tenant_id=1, server_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=true, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:82, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2024-09-26 16:01:14.072502] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:14.072509] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.075348] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A3-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_parameter, table_name.ptr()="data_size:19, data:5F5F616C6C5F7379735F706172616D65746572", ret=-5019) [2024-09-26 16:01:14.075403] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A3-0-0] [lt=54][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:14.075428] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2464][ConfigMgr][T1][YB42C0A8002D-0006230057C858A3-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:14.079736] INFO [LIB] runTimerTask (ob_work_queue.cpp:24) [2571][RSAsyncTask][T0][Y0-0000000000000000-0-0] [lt=6] add async task(this=tasktype:N9oceanbase10rootserver13ObRootService19ObRefreshServerTaskE) [2024-09-26 16:01:14.080015] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E84-0-0] [lt=6] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2024-09-26 16:01:14.080038] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E84-0-0] [lt=22][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:14.080056] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2579][RSAsyncTask0][T1][YB42C0A8002D-0006230057584E84-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:14.093032] INFO do_work (ob_rl_mgr.cpp:709) [2544][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=11] swc wakeup.(stat_period_=1000000, ready=false) [2024-09-26 16:01:14.099925] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337673867958]}) [2024-09-26 16:01:14.102622] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:14.102631] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.132723] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:14.132732] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.143331] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:291) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2024-09-26 16:01:14.143344] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:14.143352] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337674143328) [2024-09-26 16:01:14.143357] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337674043070, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:14.143375] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223403379487, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:14.143381] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:14.143389] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:14.143393] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:14.143396] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:14.143402] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.143404] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.143406] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337674143360) [2024-09-26 16:01:14.157351] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:92) [2894][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2024-09-26 16:01:14.157725] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DDA-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2024-09-26 16:01:14.157736] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DDA-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:14.157767] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2894][T1_ReqMemEvict][T1][YB42C0A8002D-0006230056B84DDA-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:14.162843] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:14.162853] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.193803] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:14.193813] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.198283] INFO [COMMON] print_io_status (ob_io_manager.cpp:1358) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=2] [IO STATUS](tenant_id=500, io_status=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 8192.00, iops: 1.00, rt: 10758.00) [2024-09-26 16:01:14.198296] INFO [COMMON] print_io_status (ob_io_manager.cpp:1408) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735904, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2024-09-26 16:01:14.198310] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS SENDER](send_index=1, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:14.198326] INFO [COMMON] print_sender_status (ob_io_struct.cpp:847) [2491][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS SENDER](send_index=2, req_count=0, reservation_ts=9223372036854775807, group_limitation_ts=9223372036854775807, tenant_limitation_ts=9223372036854775807, proportion_ts=9223372036854775807) [2024-09-26 16:01:14.200578] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337673867958]}) [2024-09-26 16:01:14.204815] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] ====== check clog disk timer task ====== [2024-09-26 16:01:14.204822] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=5] get_disk_usage(ret=0, capacity(MB):=2048, used(MB):=1945) [2024-09-26 16:01:14.204831] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [2935][T1_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=3] cannot_recycle_log_size statistics(cannot_recycle_log_size=0, threshold=644245094, need_update_checkpoint_scn=false) [2024-09-26 16:01:14.212877] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=6) [2024-09-26 16:01:14.223556] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=4] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:14.223714] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash time detail, (compute_wash_size_time=56, refresh_score_time=14, wash_time=2) [2024-09-26 16:01:14.223867] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:14.223876] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.224439] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=0, clean_num=31457) [2024-09-26 16:01:14.243455] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:14.243488] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337674243450) [2024-09-26 16:01:14.243496] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337674143360, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:14.243515] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223403479262, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:14.243529] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:14.243538] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:14.243544] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:14.243547] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:14.243554] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.243558] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.243560] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337674243499) [2024-09-26 16:01:14.254034] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=11] success to notify_need_writing_throttling True [2024-09-26 16:01:14.254043] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.269344] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B2-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2024-09-26 16:01:14.269355] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B2-0-0] [lt=10][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2024-09-26 16:01:14.269385] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2919][T1_LSMetaCh][T1][YB42C0A8002D-00062300564858B2-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:14.284129] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:14.284137] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.301143] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337673867958]}) [2024-09-26 16:01:14.301171] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:549) [2611][SysLocAsyncUp0][T0][YB42C0A8002D-0006230055A8DA32-0-0] [lt=8][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1727337674301137}) [2024-09-26 16:01:14.314249] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:14.314257] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.330790] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=19][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337674259829, now=[mts=1727337673330860]) [2024-09-26 16:01:14.330801] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=11][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337674259829, now=[mts=1727337673330860], now0=[mts=1727337673330860], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:14.330809] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=6][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926916a8c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337673330358, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:14.330828] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=18][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926916a8c0}) [2024-09-26 16:01:14.330836] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=7][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926916a8c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337673330358, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:14.330857] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=20] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c6750998, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926916a8c0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337673330358, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337672330056, use_das=false, nested_level=0, session={this:0x7f926363e038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926916a8c0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:14.330875] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:14.330879] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:14.330882] WDIAG [SQL] open (ob_result_set.cpp:157) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:14.330884] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:14.330887] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2024-09-26 16:01:14.330890] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=1, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:14.330894] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:0, client_ret:-4012}, need_retry=false) [2024-09-26 16:01:14.330901] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:14.330902] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=1][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:14.330904] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4012] failed to close result(close_ret=-4283, ret=-4012) [2024-09-26 16:01:14.330909] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C85-0-0] [lt=1][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4012, ret=-4012) [2024-09-26 16:01:14.330912] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:734) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=2] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, process_time=2000855) [2024-09-26 16:01:14.330914] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4012, ret=-4012) [2024-09-26 16:01:14.330917] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:14.330920] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1) [2024-09-26 16:01:14.330922] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4012] execute_read failed(ret=-4012, cluster_id=1, tenant_id=1) [2024-09-26 16:01:14.330924] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4012] failed to read without check sys variable(ret=-4012, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1, check_sys_variable=false, snapshot_timestamp=-1) [2024-09-26 16:01:14.330930] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4012] execute sql failed(ret=-4012, ret="OB_TIMEOUT", tenant_id=1, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2024-09-26 16:01:14.330956] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4012] load failed(ret=-4012, for_update=false) [2024-09-26 16:01:14.330960] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4012] core_table load failed(ret=-4012, ret="OB_TIMEOUT") [2024-09-26 16:01:14.330962] WDIAG [SHARE] get_baseline_schema_version (ob_global_stat_proxy.cpp:406) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4012] get failed(ret=-4012) [2024-09-26 16:01:14.330965] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_schema_service_sql_impl.cpp:880) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4012] get_baseline_schema_version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:14.330968] WDIAG [SHARE.SCHEMA] get_baseline_schema_version (ob_multi_version_schema_service.cpp:4549) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=3][errcode=-4012] get baseline schema version failed(ret=-4012, ret="OB_TIMEOUT", schema_status={tenant_id:1, snapshot_timestamp:-1, readable_schema_version:-1}) [2024-09-26 16:01:14.330971] WDIAG [SERVER] try_load_baseline_schema_version_ (ob_server_schema_updater.cpp:516) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=0] fail to update baseline schema version(tmp_ret=-4012, tmp_ret="OB_TIMEOUT", *tenant_id=1) [2024-09-26 16:01:14.331077] WDIAG [SERVER] batch_process_tasks (ob_server_schema_updater.cpp:229) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C85-0-0] [lt=2][errcode=-4023] fail to process refresh task(ret=-4023, ret="OB_EAGAIN", tasks.at(0)={type:1, did_retry:true, schema_info:{schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}}) [2024-09-26 16:01:14.331101] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:508) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] fail to batch process task(ret=-4023) [2024-09-26 16:01:14.331136] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:459) [2547][SerScheQueue1][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4023] fail to batch execute task(ret=-4023, tasks.count()=1) [2024-09-26 16:01:14.331147] INFO [SERVER] process_refresh_task (ob_server_schema_updater.cpp:255) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C86-0-0] [lt=3] [REFRESH_SCHEMA] start to process schema refresh task(ret=0, ret="OB_SUCCESS", schema_info={schema_version:-1, tenant_id:0, sequence_id:18446744073709551615}) [2024-09-26 16:01:14.331152] WDIAG [SERVER] process_refresh_task (ob_server_schema_updater.cpp:268) [2547][SerScheQueue1][T0][YB42C0A8002D-0006230058084C86-0-0] [lt=4][errcode=-4023] rootservice is not in full service, try again(ret=-4023, ret="OB_EAGAIN", GCTX.root_service_->in_service()=true, GCTX.root_service_->is_full_service()=false) [2024-09-26 16:01:14.331491] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=3][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=59) [2024-09-26 16:01:14.331499] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=8][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:14.343490] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223403579721, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:14.343501] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:14.343509] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:14.343515] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:14.343518] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:14.343524] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.343527] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.343532] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337674343474) [2024-09-26 16:01:14.345226] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:14.345236] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.345247] ERROR try_recycle_blocks (palf_env_impl.cpp:784) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95, total_unrecyclable_size_byte(MB)=1881, maximum_used_size(MB)=1945, maximum_log_stream=1, oldest_log_stream=1, oldest_scn={val:1726163155508348002, v:0}, in_shrinking=false) [2024-09-26 16:01:14.368284] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.368293] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337674368282], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:14.368300] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337674368282], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:14.368305] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:515) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts(ret=-4038, ret="OB_NOT_MASTER", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337674368282]}) [2024-09-26 16:01:14.368310] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:173) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4038] refresh gts failed(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:14.368314] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:177) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] refresh gts functor(ret=-4038, ret="OB_NOT_MASTER", gts_tenant_info={v:1}) [2024-09-26 16:01:14.375452] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:14.375461] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.394983] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1506) [2632][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=10] skip inspect bad block(last_check_time=1727334209137410, last_macro_idx=-1) [2024-09-26 16:01:14.395013] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [2645][Blacklist][T0][Y0-0000000000000000-0-0] [lt=8] blacklist_loop exec finished(cost_time=18, is_enabled=true, send_cnt=0) [2024-09-26 16:01:14.401389] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337674368282]}) [2024-09-26 16:01:14.405678] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=30] success to notify_need_writing_throttling True [2024-09-26 16:01:14.405686] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.412959] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=5) [2024-09-26 16:01:14.424766] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:14.424869] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=12] Wash time detail, (compute_wash_size_time=95, refresh_score_time=15, wash_time=1) [2024-09-26 16:01:14.425664] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=31457, clean_num=31457) [2024-09-26 16:01:14.435846] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=32] success to notify_need_writing_throttling True [2024-09-26 16:01:14.435855] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.441379] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1766) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=9][errcode=-4012] get gts fail(ret=-4012, expire_ts=1727337700364812, now=[mts=1727337673440462]) [2024-09-26 16:01:14.441387] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1780) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=8][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1727337700364812, now=[mts=1727337673440462], now0=[mts=1727337673440462], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2024-09-26 16:01:14.441402] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:623) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=13][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x7f926902cea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337673440357, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:14.441421] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:726) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=17][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:1}, session={this:0x7f9269128038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926902cea0}) [2024-09-26 16:01:14.441429] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:554) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=8][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x7f926902cea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337673440357, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}) [2024-09-26 16:01:14.441437] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:622) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=7] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x7f92c56d2378, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, snapshot_acquire_addr:"0.0.0.0:0", parts:[]}, savepoint=0, tx_desc={this:0x7f926902cea0, tx_id:{txid:0}, state:1, addr:"192.168.0.45:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:-1, tx_consistency_type:0, isolation:-1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:1, alloc_ts:1727337673440357, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:-1, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:-1, cluster_version:0, flags_.SHADOW:true, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, ref:1}, plan_type=1, stmt_type=1, has_for_update=false, query_start_time=1727337670434817, use_das=false, nested_level=0, session={this:0x7f9269128038, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x7f926902cea0}, plan=0x7f92ae448050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2024-09-26 16:01:14.441455] WDIAG [SQL] start_stmt (ob_result_set.cpp:354) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=17][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:1, schema_version:0, object_type:1, is_db_explicit:false, is_existed:true}]) [2024-09-26 16:01:14.441459] WDIAG [SQL] do_open_plan (ob_result_set.cpp:539) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4283] fail start stmt(ret=-4283) [2024-09-26 16:01:14.441476] WDIAG [SQL] open (ob_result_set.cpp:157) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=16][errcode=-4283] execute plan failed(ret=-4283) [2024-09-26 16:01:14.441479] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2024-09-26 16:01:14.441482] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:768) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}) [2024-09-26 16:01:14.441486] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, retry_cnt=3, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:14.441491] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3] will sleep(sleep_us=3000, remain_us=25993322, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4283, timeout_timestamp=1727337700434812) [2024-09-26 16:01:14.443572] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:14.443588] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337674443568) [2024-09-26 16:01:14.443594] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337674243499, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:14.443611] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223403678941, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:14.443617] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:14.443624] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:14.443629] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:14.443631] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:14.443637] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.443653] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.443655] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337674443598) [2024-09-26 16:01:14.444569] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:3, local_retry_times:3, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true) [2024-09-26 16:01:14.444581] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=8][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:14.444584] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4283] result set close failed(ret=-4283) [2024-09-26 16:01:14.444586] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2024-09-26 16:01:14.444591] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_schema_status' ORDER BY row_id, column_name"}, record_ret=-4283, ret=-4283) [2024-09-26 16:01:14.444630] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=74) [2024-09-26 16:01:14.444633] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:14.465890] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:14.465897] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.471184] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=3] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.471192] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=8][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.471197] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=4][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.471201] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.471203] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.471205] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.471829] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.471832] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.471834] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.471838] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.471841] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.471843] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.472413] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.472416] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.472418] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.472421] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.472423] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.472425] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.473032] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.473035] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.473037] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.473040] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.473042] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.473044] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.473596] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=12] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.473601] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.473603] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.473606] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.473608] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.473609] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.474179] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.474182] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.474184] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.474187] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.474189] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.474191] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=1][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.474853] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.474857] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.474860] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=3][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.474864] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.474866] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.474868] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.475410] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.475415] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.475417] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.475421] INFO [STORAGE.TRANS] get_number (ob_timestamp_access.cpp:48) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.475422] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.475424] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:234) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2][errcode=-4023] get_gts_from_local_timestamp_service fail(leader="192.168.0.45:2882", server="192.168.0.45:2882", tmp_ret=-4038) [2024-09-26 16:01:14.476015] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.476020] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.476654] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.476662] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=1][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.477995] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=24][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.478001] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.478026] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:333) [2568][LSSysTblUp0][T0][YB42C0A8002D-00062300558D4746-0-0] [lt=9][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1727336098526593}) [2024-09-26 16:01:14.478970] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.478976] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.480956] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.480971] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.481932] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=14][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.481931] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.482883] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.482925] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.483411] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.483492] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.484856] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [2490][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=12][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:206, tid:3001}]) [2024-09-26 16:01:14.484871] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.484950] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.485448] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.485475] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.485980] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.486202] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.486784] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.486790] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.487754] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.487762] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.488644] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=35][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.488653] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.489235] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.489235] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.489757] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.489771] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.490697] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.490698] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.491675] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.491695] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.492230] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.492236] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=9][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.492883] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.492897] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.493466] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.493496] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.493983] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.494018] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=2][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.494547] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.494553] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.495466] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.495476] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.495930] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:14.495937] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.496018] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.496043] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.496571] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.496828] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.497445] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.497491] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.498226] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.498240] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.498981] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.499015] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.499547] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.499937] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.500354] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.500792] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.501113] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.501381] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.501392] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337674368282]}) [2024-09-26 16:01:14.501691] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.501998] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.502248] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.502611] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.502821] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.503172] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.503393] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.503709] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.503935] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.504409] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.504480] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.505017] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=10][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.505070] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.505588] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=7][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.505657] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.506109] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.506240] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=5][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.506696] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.506808] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=4][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.507243] WDIAG [STORAGE.TRANS] get_gts_from_local_timestamp_service_ (ob_gts_source.cpp:294) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=3][errcode=-4038] global_timestamp_service get gts fail(leader="192.168.0.45:2882", tmp_gts=0, ret=-4038, ret="OB_NOT_MASTER") [2024-09-26 16:01:14.526033] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:14.526041] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.526438] WDIAG [COORDINATOR] get_ls_election_reference_info (ob_leader_coordinator.cpp:175) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594D-0-0] [lt=36][errcode=-4018] can not find this ls_id in all_ls_election_reference_info_(ret=-4018, ret="OB_ENTRY_NOT_EXIST", ls_id={id:1}, all_ls_election_reference_info=[]) [2024-09-26 16:01:14.526447] WDIAG [COORDINATOR] refresh_ (election_priority_v1.cpp:225) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594D-0-0] [lt=9][errcode=-4018] fail to get ls election reference info(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, *this={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:14.526457] WDIAG [COORDINATOR] operator() (election_priority_impl.cpp:252) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594D-0-0] [lt=9][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id_={id:1}, element={is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}) [2024-09-26 16:01:14.526462] WDIAG [COORDINATOR] refresh (election_priority_impl.cpp:268) [3536][T1_L0_G2][T1][YB42C0A8002D-000623005718594D-0-0] [lt=5][errcode=-4018] refresh priority failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", MTL_ID()=1, ls_id={id:1}, *this={priority:{is_valid:false, is_observer_stopped:false, is_server_stopped:false, is_zone_stopped:false, fatal_failures:[{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}], is_primary_region:false, serious_failures:[{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}], is_in_blacklist:false, in_blacklist_reason:, scn:{val:0, v:0}, is_manual_leader:false, zone_priority:9223372036854775807}}) [2024-09-26 16:01:14.539929] INFO eloop_run (eloop.c:143) [2531][pnio1][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1727337674539927, bytes: 14758445, bw: 0.002939 MB/s, add_ts: 1007596, add_bytes: 3105 [2024-09-26 16:01:14.543692] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:14.543703] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337674543688) [2024-09-26 16:01:14.543708] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337674443597, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:14.543715] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:780) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] current server is WRS leader, need start CLUSTER weak read service(tenant_id=1, serve_leader_epoch=0, cur_leader_epoch=374, cluster_service_tablet_id_={id:226}, in_service=false, can_update_version=false, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:14.543723] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:360) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] begin start service(tenant_id=1, is_in_service()=false, can_update_version=false) [2024-09-26 16:01:14.543726] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:362) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] start TenantWeakReadClusterService(tenant_id=1) [2024-09-26 16:01:14.544057] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, table_name.ptr()="data_size:23, data:5F5F616C6C5F7765616B5F726561645F73657276696365", ret=-5019) [2024-09-26 16:01:14.544065] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=6][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_weak_read_service, ret=-5019) [2024-09-26 16:01:14.544068] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=3][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_weak_read_service, db_name=oceanbase) [2024-09-26 16:01:14.544071] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_weak_read_service) [2024-09-26 16:01:14.544075] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=3][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:14.544077] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:14.544080] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_weak_read_service' doesn't exist [2024-09-26 16:01:14.544082] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=1][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:14.544085] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:14.544086] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:14.544088] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:14.544090] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:14.544093] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:14.544095] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:14.544101] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:14.544105] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=4][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:14.544109] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=3][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:14.544111] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=1][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:14.544113] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=1][errcode=-5019] fail to handle text query(stmt=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '', ret=-5019) [2024-09-26 16:01:14.544115] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:14.544117] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:14.544139] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=20][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:14.544145] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=5][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:14.544147] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:14.544149] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:14.544155] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=1][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:14.544158] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:14.544160] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2932][T1_TenantWeakRe][T1][YB42C0A8002D-0006230057885792-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = ''"}, aret=-5019, ret=-5019) [2024-09-26 16:01:14.544163] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:14.544165] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:14.544167] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:14.544169] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8620, start=1727337674543984, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:14.544172] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:14.544175] WDIAG [STORAGE.TRANS] query_cluster_version_range_ (ob_tenant_weak_read_cluster_service.cpp:209) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql read fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, tenant_id=1, sql=select min_version, max_version from __all_weak_read_service where tenant_id = 1 and level_id = 0 and level_value = '') [2024-09-26 16:01:14.544207] WDIAG [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:391) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] query cluster version range from WRS table fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:14.544210] INFO [STORAGE.TRANS] start_service (ob_tenant_weak_read_cluster_service.cpp:451) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] start service done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, in_service=false, leader_epoch=0, current_version={val:0, v:0}, delta=1727337674544209, min_version={val:0, v:0}, max_version={val:0, v:0}, max_stale_time=5000000, all_valid_server_count=0, total_time=490, wlock_time=9, check_leader_time=0, query_version_time=0, persist_version_time=0) [2024-09-26 16:01:14.544217] WDIAG [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:823) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] start CLUSTER weak read service fail(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2024-09-26 16:01:14.544219] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:833) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, need_start_service=true, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=374, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2024-09-26 16:01:14.544237] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] get wrs ts(ls_id={id:1}, delta=223403780280, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:14.544242] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:14.544248] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:14.544252] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:14.544254] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:14.544260] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.544262] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.544264] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337674544223) [2024-09-26 16:01:14.551978] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:222) [2931][T1_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=45] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=32212254, evict_high_size_level=90000, evict_batch_count=0, elapse_time=0, size_used=6747, mem_used=12478464) [2024-09-26 16:01:14.556124] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:14.556131] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.568786] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:539) [2938][T1_LSArchiveMgr][T1][YB42C0A8002D-0006230055C84B24-0-0] [lt=6] gc stale ls task succ [2024-09-26 16:01:14.586436] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=10] success to notify_need_writing_throttling True [2024-09-26 16:01:14.586444] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.602095] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=4] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337674368282]}) [2024-09-26 16:01:14.602129] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2667][RpcIO][T0][Y0-0000000000000000-0-0] [lt=6] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.602141] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2668][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.602148] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2669][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.602154] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2666][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.613042] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=4) [2024-09-26 16:01:14.616588] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:14.616598] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.625949] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:14.626139] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash time detail, (compute_wash_size_time=142, refresh_score_time=16, wash_time=17) [2024-09-26 16:01:14.627254] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=5] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=62914, clean_num=31457) [2024-09-26 16:01:14.637914] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:139) [2736][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=5] doing detect recover operation(events_with_ops=[{event:{type:RESOURCE NOT ENOUGH, module:LOG, info:clog disk full event, level:FATAL}}, {event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2024-09-26 16:01:14.643779] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:14.643791] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337674643775) [2024-09-26 16:01:14.643797] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337674543711, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:14.643814] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] get wrs ts(ls_id={id:1}, delta=223403879790, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:14.643819] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:14.643827] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:14.643832] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:14.643835] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:14.643840] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.643842] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.643843] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337674643800) [2024-09-26 16:01:14.646711] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] success to notify_need_writing_throttling True [2024-09-26 16:01:14.646718] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.668104] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:189) [2948][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2024-09-26 16:01:14.676830] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:14.676837] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.702322] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2547][SerScheQueue1][T1][YB42C0A8002D-0006230058084C86-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337674368282]}) [2024-09-26 16:01:14.706920] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:14.706929] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.737819] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:14.737828] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.743839] WDIAG [PALF] convert_to_ts (scn.cpp:265) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2024-09-26 16:01:14.743848] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:541) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1727337674743838, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1727337674743838, force_self_check=false, weak_read_refresh_interval=100000) [2024-09-26 16:01:14.743866] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:14.743873] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337674743864) [2024-09-26 16:01:14.743876] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337674643799, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:14.743892] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get wrs ts(ls_id={id:1}, delta=223403980302, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:14.743899] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:14.743907] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:14.743911] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:14.743914] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:14.743920] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.743922] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.743924] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337674743879) [2024-09-26 16:01:14.761874] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:83) [2928][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=4] XA scheduler heartbeat task statistics(avg_time=0) [2024-09-26 16:01:14.767905] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:14.767912] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=5][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.785104] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=4] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2024-09-26 16:01:14.785115] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=8] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2024-09-26 16:01:14.785121] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:228) [2936][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=4] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=695) [2024-09-26 16:01:14.798069] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:14.798079] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.802612] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2582][RSAsyncTask3][T1][YB42C0A8002D-0006230057684E63-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337674368282]}) [2024-09-26 16:01:14.813111] INFO [COMMON] replace_map (ob_kv_storecache.cpp:746) [2514][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=15728, map_replace_skip_count_=3) [2024-09-26 16:01:14.820862] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2678][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=7] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.820937] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2679][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=5] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.821170] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2682][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.821405] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2671][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.821412] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2680][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.821417] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [2677][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.821435] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2672][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.822024] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2674][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.822033] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2675][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.822342] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [2673][BatchIO][T0][Y0-0000000000000000-0-0] [lt=4] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.822353] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [2684][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=3] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2024-09-26 16:01:14.827529] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1157) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=2252308480, global_cache_size=12484608, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 1]) [2024-09-26 16:01:14.827572] ERROR period_calc_disk_usage (palf_env_impl.cpp:1349) [2819][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4264] Log out of disk space(msg="log disk space is almost full", ret=-4264, total_size(MB)=2048, used_size(MB)=1945, used_percent(%)=95, warn_size(MB)=1638, warn_percent(%)=80, limit_size(MB)=1945, limit_percent(%)=95) [2024-09-26 16:01:14.827645] INFO [COMMON] wash (ob_kvcache_store.cpp:342) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11] Wash time detail, (compute_wash_size_time=58, refresh_score_time=15, wash_time=2) [2024-09-26 16:01:14.828333] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:646) [2513][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=0, clean_start_pos=94371, clean_num=31457) [2024-09-26 16:01:14.828405] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:14.828410] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=4][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.843936] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:452) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="192.168.0.45:2882", cluster_service_tablet_id={id:226}) [2024-09-26 16:01:14.843951] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:869) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1727337674843933) [2024-09-26 16:01:14.843956] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:879) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1727337674743879, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2024-09-26 16:01:14.843971] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] get_max_applied_scn(scn={val:1727114270763524000, v:0}, this={ls_id_:{id:1}, role_:1, proposal_id_:374, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:1727114270763524000, v:0}, max_applied_cb_scn_:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:14.843992] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get_min_unreplayed_log_info(lsn={lsn:629872191939}, scn={val:1727114270763524001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:1, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1727336083532153, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:36054089, ret_code_:0}}, next_to_submit_lsn_:{lsn:629872191939}, next_to_submit_scn_:{val:1727114270763524001, v:0}, base_lsn_:{lsn:627966418944}, base_scn_:{val:1726163586617842000, v:0}, iterator_:{iterator_impl:{buf_:0x7f929f205000, next_round_pread_size:3678208, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:629872191939}, end_lsn:{lsn:629872191939}, read_buf:{buf_len_:3682304, buf_:0x7f929f205000}, block_size:67104768, log_storage_:0x7f92b3ff7470}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:18446744073709551615, v:3}, curr_entry:{LogEntryHeader:{magic:0, version:0, log_size:-1, scn_:{val:18446744073709551615, v:3}, data_checksum:0, flag:0}}, init_mode_version:0, accumulate_checksum:-1, curr_entry_is_padding:0, padding_entry_size:0, padding_entry_scn:{val:18446744073709551615, v:3}}}}}) [2024-09-26 16:01:14.844010] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:188) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=223404079748, log_service_ts=1727114270763524, min_tx_service_ts=4611686018427387, timestamp={val:1727114270763524000, v:0}) [2024-09-26 16:01:14.844017] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1727114269748842003, v:0}", last_lsn={lsn:629872191028}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2024-09-26 16:01:14.844024] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:107) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1727114270763524000, v:0}}) [2024-09-26 16:01:14.844028] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:683) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2024-09-26 16:01:14.844031] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:639) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2024-09-26 16:01:14.844037] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:83) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.844040] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:317) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2024-09-26 16:01:14.844042] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:597) [2932][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f92d36cc690, server_version_epoch_tstamp_=1727337674843959) [2024-09-26 16:01:14.858502] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7] success to notify_need_writing_throttling True [2024-09-26 16:01:14.858509] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=6][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.868682] INFO [STORAGE.TRANS] handle_request (ob_timestamp_access.cpp:32) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=1] ObTimestampAccess service type is FOLLOWER(ret=-4038, service_type=0) [2024-09-26 16:01:14.868691] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:226) [2665][TsMgr][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4038] post local gts request failed(ret=-4038, ret="OB_NOT_MASTER", server="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337674868678], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:14.868699] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:561) [2665][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4038] post gts request failed(ret=-4038, ret="OB_NOT_MASTER", leader="192.168.0.45:2882", msg={tenant_id:1, srr:[mts=1727337674868678], range_size:1, sender:"192.168.0.45:2882"}) [2024-09-26 16:01:14.881803] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:7504) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2] table not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, table_name.ptr()="data_size:16, data:5F5F616C6C5F6D657267655F696E666F", ret=-5019) [2024-09-26 16:01:14.881825] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:7462) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=20][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_merge_info, ret=-5019) [2024-09-26 16:01:14.881830] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:7307) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=4][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_merge_info, db_name=oceanbase) [2024-09-26 16:01:14.881833] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:7152) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=3][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_merge_info) [2024-09-26 16:01:14.881837] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:2069) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] fail to resolve table(ret=-5019) [2024-09-26 16:01:14.881840] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:2124) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] fail to resolve sys view(ret=-5019) [2024-09-26 16:01:14.881843] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:2220) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=1][errcode=-5019] Table 'oceanbase.__all_merge_info' doesn't exist [2024-09-26 16:01:14.881846] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:13031) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2024-09-26 16:01:14.881848] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:12955) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=1][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2024-09-26 16:01:14.881849] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:2624) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=1][errcode=-5019] resolve basic table failed(ret=-5019) [2024-09-26 16:01:14.881852] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3644) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2024-09-26 16:01:14.881854] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1095) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2024-09-26 16:01:14.881857] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1297) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] resolve normal query failed(ret=-5019) [2024-09-26 16:01:14.881859] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:183) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=1][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2024-09-26 16:01:14.881865] WDIAG [SQL] generate_stmt (ob_sql.cpp:2954) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] failed to resolve(ret=-5019) [2024-09-26 16:01:14.881868] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3074) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:14.881871] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:4903) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2024-09-26 16:01:14.881873] WDIAG [SQL] handle_text_query (ob_sql.cpp:2649) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] fail to handle physical plan(ret=-5019) [2024-09-26 16:01:14.881876] WDIAG [SQL] stmt_query (ob_sql.cpp:216) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_merge_info WHERE tenant_id = '1', ret=-5019) [2024-09-26 16:01:14.881878] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:755) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] executor execute failed(ret=-5019) [2024-09-26 16:01:14.881880] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:913) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2024-09-26 16:01:14.881886] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:935) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=4][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2024-09-26 16:01:14.881895] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:222) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=7][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:14.881898] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:202) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=3][errcode=-5019] result set close failed(ret=-5019) [2024-09-26 16:01:14.881899] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:918) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=1][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2024-09-26 16:01:14.881906] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:948) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, record_ret=-5019, ret=-5019) [2024-09-26 16:01:14.881909] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:350) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2024-09-26 16:01:14.881911] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:984) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885090-0-0] [lt=2][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1'"}, aret=-5019, ret=-5019) [2024-09-26 16:01:14.881914] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1768) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:01:14.881916] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1043) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2024-09-26 16:01:14.881918] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1708) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1, tenant_id=1) [2024-09-26 16:01:14.881921] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=2][errcode=-5019] query failed(ret=-5019, conn=0x7f92cefa8900, start=1727337674881731, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:01:14.881924] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-5019] read failed(ret=-5019) [2024-09-26 16:01:14.881926] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:49) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, meta_tenant_id=1, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1') [2024-09-26 16:01:14.881957] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:890) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-5019] failed to load global merge info(ret=-5019, ret="OB_TABLE_NOT_EXIST", global_merge_info={tenant_id:1, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2024-09-26 16:01:14.881970] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:1005) [2892][T1_FreInfoReloa][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] fail to refresh merge info(tmp_ret=-5019, tmp_ret="OB_TABLE_NOT_EXIST") [2024-09-26 16:01:14.882222] WDIAG [STORAGE.TRANS] get_task (ob_ts_mgr.cpp:287) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885091-0-0] [lt=2][errcode=-4023] ObTsSyncGetTsCbTaskPool failed to get task(ret=-4023, thread_id=279) [2024-09-26 16:01:14.882227] WDIAG [STORAGE.TRANS] get_gts_sync (ob_ts_mgr.cpp:946) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885091-0-0] [lt=5][errcode=-4023] failed to get ObTsSyncGetTsCbTask, fall back to sleep(ret=-4023) [2024-09-26 16:01:14.888657] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=8] success to notify_need_writing_throttling True [2024-09-26 16:01:14.888666] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2024-09-26 16:01:14.902875] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:579) [2892][T1_FreInfoReloa][T1][YB42C0A8002D-0006230056885091-0-0] [lt=2] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=1727337674868678]}) [2024-09-26 16:01:14.918772] INFO [PALF] notify_need_writing_throttling (log_io_worker_wrapper.cpp:121) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] success to notify_need_writing_throttling True [2024-09-26 16:01:14.918780] WDIAG [PALF] recycle_blocks_ (palf_env_impl.cpp:1145) [2817][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=7][errcode=0] there is not any block can be recycled, need verify the baselsn of PalfHandleImpl whether has been advanced(ret=0, this={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"192.168.0.45:2882", log_dir:"/root/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):2048, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):1881, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}})